Difference between revisions of "Future Development Ideas"

From EPICSWIKI
 
(84 intermediate revisions by 3 users not shown)
Line 1: Line 1:
These are my notes for what I think we can/should add to V3 through an evolutionary process.  The ordering is probably vaguely right, but not necessarily that accurate.
These are projects that can/should be added to EPICS V3 using an evolutionary process.  The ordering is not necessarily accurate.




Line 5: Line 5:




== 3.15 Cleanup ==
== Tasks for the 2014 Codeathon ==


3.15 already includes the Up change to the build system; the ramifications of this change have not been followed up completely though:
These items are of particular interest to have completed before the 3.15.0.2 release in September 2014.


* Provide a perl script for converting 3.14 apps to 3.15
<blockquote>
* Bring over latest templates from 3.14
* Implications for building extensions not worked out
* makeBaseExt.pl is probably no longer needed
* Install RULES files into a top level directory
* Change script installation rules (see ctlsys)
* Version numbering - switch from alpha/beta to release candidates RC1 etc. which probably needs some change to <tt>CONFIG_BASE_VERSION</tt>


=== Bug Triage and Fixing ===


== PV Aliases ==
See the [https://bugs.launchpad.net/epics-base Launchpad bug tracker]. Bug
triage involves understanding and replicating the bug and ensuring that it still exists in the latest release. Bugs tagged 'codeathon' are especially appropriate for fixing as Codeathon projects.


The ability to add aliases to existing records has already been demonstrated, although the implementation that goes into Base might be somewhat different.
=== Review Merge Proposals ===


The [https://code.launchpad.net/epics-base/+activereviews Launchpad active review] page lists branches which have been proposed for merging.


== Compiling DBD Files ==
=== Record Reference Manual ===


My Perl implementation of the DBD file parser is still in process, and I'm hoping to revive this development effort to convert all DBD file information into C tables that get compiled into the IOC executable.
Work has started on embedding the RRM text inside the record type DBD files, but most record types have still to be documented this way. See ANJ's [https://code.launchpad.net/~anj/epics-base/dbd2html DBD To HTML branch] which contains the work done so far.


With this parser now being written in Perl, it will become easier to extend the facilities that we can implement, such as adding a new field type...
=== Dynamic Array Support ===


Neither the CAS nor the PV Gateway support the dynamic array size protocol changes that were added in 3.14.12, and it's starting to show.


== Unbundling ==
=== More test programs ===


Split Base into three or more independent modules.  Each module would have its own version number, and the modules can therefore each have an independent existance, although there would obviously be specific dependencies associated with particular versions. I see the modules as follows:
There are test programs in libCom/test which cover quite a lot of the libCom functionality, but they are by far from complete.  Writing test code is a very good way to learn the capabilities of the code being tested, and is not usually hard to to, so this would be a good "getting started" project. Specific tests include:


=== Core ===
* Tests for each of the Base record types and device support. This should be based on ANJ's [https://code.launchpad.net/~epics-core/epics-base/ioc-test-module/ IOC Test Module] branch so the tests can be written in Perl.
* Any parts of libCom which don't currently have tests in libCom/test (C code)
* Rewrite the tests from the epics-soft-test suite (C or Perl).


These are the common parts used by everything:
=== Convert makeBpt into Perl ===


* Build system
The makeBpt tool in src/bpt generates non-linear break-point tables from data files.  A fair bit of the C code is concerned with parsing the input file.  There is no particular need for this code to be written in C, speed is not an issue, and eventually I hope to convert all programs needed for building EPICS into Perl code.  Maybe start this project by writing a program to test the output, then convert the C code into Perl.
* OSI layer
* libCom
* Antelope
* Flex
* makeApp tool


=== Channel Access ===
=== Support for 64-bit integer fields ===


These are Jeff Hill's responsibility.  He may want to further unbundle GDD and CAS, which is up to him.  
ANJ has started adding support for DBF_INT64 and DBF_UINT64 types to the database, but the result has not been heavily tested and may not work on all target architectures (in particular VxWorks). See ANJ to work on this.


* CA-specific build rules (probably none)
</blockquote>
* CA client library
* CA Tools
* GDD
* The CAS server tool
* CA and CAS application makeApp templates


=== IOC ===
== Other Tasks which could be done now ==


The rest is code that runs on the IOC.
These items can be worked on immediately as they do not depend on any other work to have been completed.


* IOC-specific build rules
<blockquote>
* Static and runtime database access
* Runtime database
* RSRV (the old CA server code)
* Standard record types
* Soft device support
* The registry
* IOC shell
* IOC application makeApp templates


=== Bug Triage and Fixing ===


== Redundancy ==
See the [https://bugs.launchpad.net/epics-base Launchpad bug tracker]. Bug
triage involves understanding and replicating the bug and ensuring that it still exists in the latest release. Bugs tagged 'codeathon' are especially appropriate for fixing as Codeathon projects.


DESY funded project for Bob & John to add redundant control of remote I/O, allowing hot-spare backup IOCs.
=== Review Merge Proposals ===


The [https://code.launchpad.net/epics-base/+activereviews Launchpad active review] page lists branches which have been proposed for merging.


== CA Event Filtering ==
=== Documentation ===


LANSCE wants Time and Flavored Data.  I think this should be implemented differently to how Jeff is currently planning, with my design involving much simpler changes to the core code but requiring a bit more applications work from the LANL software guys.
Lots we can do here:


The main change would be to allow a CA client to subscribe to monitors on a channel but include another CHID in the subscription giving a channel that controls whether an individual event would be sent to that client or not.  The conditional channel would give a boolean value, which would be checked every time a monitor event is notified to the server; if true the server sends it to this client, if false it doesn't.  Initially that channel would have to be on the same IOC as the main channel, but if Jeff wants to implement a suitable CA client in the server that restriction could be removed.
<blockquote>


To implement the Time and Flavored Data that they want LANSCE would have to create records (or record fields) in the IOCs that become true when their events are occurring and false otherwise. A client that wants to configure a non-standard flavor might have to ask a central facility to allocate and set up an appropriate calc record on its behalf.
==== Application Developers Guide ====


The major advantage of this approach is that it only needs changes in the CA server and client code; there are no modifications needed to the database, record or device support layers at all.  Other sites will be able to make use of the new CA facility as well, whereas Jeff's original requirement was very LANL-specific and would be unlikely to be used much elsewhere.
* Write a "Building and Porting EPICS" chapter
* Restructure the AppDevGuide to make it more accessible
** Combine the libCom and libCom/osi chapters
** Maybe have a separate section for all callable APIs
** Consider moving API documentation into the appropriate source files and using Doxygen or perldoc to extract it.


==== Other Documents ====


== Variable Length Strings ==
* Look at all the documentation we have and see what's missing


I'm already adding my new BaseString C++ classes to libCom, and I intend to create a set of C wrapper routines for the base class operations.  This can form the basis of a new string type for use in the database.
=== VxWorks SMP Support ===


Recent versions of vxWorks (6.8+) support SMP but present a slightly different API that removes intLock() and taskLock(), and replaces taskVar with __thread variables.  The current OSD routines for vxWorks will need significant modification to work with the new APIs.  This will probably require creating a new OS-Class.


== Alternate DB File Formats ==
Can obviously only be done by someone with access to vxWorks 6.8 or later.


There are calls to replace the existing DB file format with an XML-based format.  Currently the DB file parser is too highly integrated into the process of record creation to be able to add a second format, but it would be feasible to separate these two and hence allow an expat-based or similar parser to be written.  This would probably require a close analysis and upgrade to the locking arrangements of the database.
=== Extended Device Support ===


Experience converting device support layers to Extended support exposed the need to clean up the extended interface, i.e. reduce the code duplication which is currently required since the record knows nothing about the device change.


== Online Record Creation ==
Libs affected: db, rec.


With good database locking, it might be feasible to look at the ability to add new record instances at runtime, and maybe delete as well.  This would probably need a change to the record interface to initialize the new instances.
=== More flexible Access Security ===


The AS design defines each field's security level in the record description, and limits us to only levels 0 and 1.  Some sites may want finer control over accesses: being able to put individual fields of a specific record into a different level, and having additional static levels (use a bitmask?) would help in a number of situations. It is not clear how much work this would entail though.


== Dynamically Loaded Modules ==
Priority: Low<br>
Libs affected: dbStatic, db, as, ...


After converting the DBD load process into table registration, it should become easier to allow support modules (containing record types, device support and other drivers) to be discovered and loaded dynamically.
=== Authentication ===


From Ralph: Two or three institutes need a way to password-protect access to certain databases. The usual encryption libraries are freely available, but I guess this issue needs a lot more thinking. Granularity: by record, by connection = IOC, or centralised? Should the authentication last for the client's lifetime or should a short form (some security token or key) be sent with every CA data package?


== ... ==
=== Secure CA ===
 
Secure channel access protocol - add SSL wrappers around CA's TCP connections to make them secure.
 
Libs affected: ca, cas, rsrv
 
</blockquote>
 
== Things which can't be completed yet ==
 
These are tasks are waiting on other people or for other tasks to be completed before they can be finished, although there may be infrastructure to support them that can be worked on in parallel.
 
<blockquote>
 
=== IOCSH Command Registration ===
 
Add DBD file syntax to convert DBD file information into a C header file which declares the command functions (similar to how we do record types in the current code-base).  This header would then be included by the C file that defines those functions, thus the C compiler would do the work of ensuring that the routine implemented actually matches the command that will call it.
 
The command information in the DBD file will also be useful for future versions of the IRMIS parser for st.cmd scripts, where we need to be able to tie the device support configuration commands into IRMIS' need to map hardware devices to record addresses.
 
Libs affected: dbHost, all libs Registrar code.
 
=== Link Support ===
 
See original proposal by ANJ, which needs to be modified to add notify interfaces. This work is already under way, talk to ANJ.
 
Part of this development (June 2016) includes [[JSON Field Values]].
 
=== Array Field Support ===
 
Make it easier to add array fields to record types without having to do as much coding in the record support.  See [http://www.aps.anl.gov/epics/core-talk/2007/msg00078.php Ben Franksen email] and the work done by Nick Rees at the 2009 Codeathon.
 
Libs affected: dbHost, dbStatic, db, rec.
 
=== Variable Length Strings ===
 
The BaseString C++ APIs and classes were developed on the cvs-trunk branch and are being used by Jeff Hill in his replacement IOC server. This interface can form the basis of a new variable length string type for use in the database in addition to DBF_STRING.
 
Libs affected: dbHost, dbStatic, db, ...
 
=== Alternate DB File Formats ===
 
There have been calls to be able to replace the existing DB file format with an XML-based format. Currently the DB file parser is too highly integrated into the process of record creation to easily add a second format, but it would be feasible to separate these two and hence allow an expat-based or similar parser to be written. This would probably require a close analysis and upgrade to the locking arrangements of the database.  Alternative to XML would be JSON, for which a parser is already being included in the jsonfm branch.
 
Libs affected: dbHost?, dbStatic.
 
=== New Record API ===
 
These requests all need a significant change to the record support API.  They are also fairly closely related, so should be used as input into any new API that gets devised.
 
<blockquote>
 
==== Asynchronous Gets (3.16++) ====
 
From Tim Mooney: It would be great if an input link could initiate processing, and get a value that is known to be the result of that processing, even if
records to be processed are asynchronous.  Currently, a record that needs completion-qualified data must know how to trigger the processing that generates that data.
 
From Ben Franksen: Implemented an asyncronous dbGetLinkCallback() routine.  To do this properly though needs an upgrade to the record support API, but maybe could be shoe-horned into the the existing API by adding a dbGetLinkCallback() with a private callback pointer, and we'd require new record types to use this.  What does this do to the device support interface though?  Implications need study.
 
Andrew's Comment: This should be part of the implementation of linkSupport which I proposed in detail many years ago.  The purpose of the "Compiling DBD Files" item above is to make extending the DBD file syntax easier to do in the future, which is a necessary part of that development.
 
==== Blocking Puts (3.16++) ====
 
From Tim Mooney: It would be nice to have a link attribute that could put, request processing, and wait for completion.  We have this capability now, in dbCaPutCallback(), but it's selectable only at DCT time.
 
The idea is for a version of the SEQ record where rather than a fixed delay, the second link would wait for the completion callback from the first link.  This should be possible today with a special record type, but Tim suggests it should be possible with any link field of any record type.
 
Andrew's Comment: This should be part of the implementation of linkSupport which I proposed in detail many years ago.  The purpose of the "Compiling DBD Files" item above is to make extending the DBD file syntax easier to do in the future, which is a necessary part of that development.
 
 
==== Multiple Master problem (3.16++) ====
 
From Jeff: Changes to output values in a PLC or similar device initiated outside of the output record. This could be resolved adding a new routine to the RSET that the device support calls back when it has a new raw ouptut value to back-propagate to the VAL field.  Currently done by some devices that set PACT and call the record's process() routine to use its second-half processing only.
 
 
==== Records Can't Block (3.16++) ====
 
From Tim Mooney: I think almost half the code in the sscan record is devoted to
bookmarking. In the middle of some algorithm, the record needs to wait for external events; the only way back in is process(), and so the record has to
have a complicated arrangement of switches that get it back into the
algorithm it was executing, with enough state information to decide what to do
next. At the same time, the record must check to see if it's now processing
because of some out-of-band request from the user.  Normal and exceptional processing must, therefore, be mixed together in the switch
arrangement, and adding new behaviors to the record becomes a very hairy undertaking.
 
But what the record would really like to do is just wait with its
context intact for events, and allow special() to release it from the wait
if the user has requested something that requires this.  I guess this means the record should include author-defined event types in the wait mask, and special() should be able to generate author-defined events.
 
 
</blockquote>
 
=== Dynamically Loaded Modules (3.16++) ===
 
After converting the DBD load process into table registration, it should become easier to allow support modules (containing record types, device support and other drivers) to be discovered and loaded dynamically.  Unloading is a very different matter though, we don't have the shutdown facilities to allow that.
 
=== Atomic Operations through CA (3.16++) ===
 
This is related to the Asynch Gets issue above; implement Read-Write (Swap, permits semaphores), Process-Read and Write-Process-Read (Command/Response) as atomic operations available through Channel Access.
 
</blockquote>

Latest revision as of 19:51, 1 June 2016

These are projects that can/should be added to EPICS V3 using an evolutionary process. The ordering is not necessarily accurate.



Tasks for the 2014 Codeathon

These items are of particular interest to have completed before the 3.15.0.2 release in September 2014.

Bug Triage and Fixing

See the Launchpad bug tracker. Bug triage involves understanding and replicating the bug and ensuring that it still exists in the latest release. Bugs tagged 'codeathon' are especially appropriate for fixing as Codeathon projects.

Review Merge Proposals

The Launchpad active review page lists branches which have been proposed for merging.

Record Reference Manual

Work has started on embedding the RRM text inside the record type DBD files, but most record types have still to be documented this way. See ANJ's DBD To HTML branch which contains the work done so far.

Dynamic Array Support

Neither the CAS nor the PV Gateway support the dynamic array size protocol changes that were added in 3.14.12, and it's starting to show.

More test programs

There are test programs in libCom/test which cover quite a lot of the libCom functionality, but they are by far from complete. Writing test code is a very good way to learn the capabilities of the code being tested, and is not usually hard to to, so this would be a good "getting started" project. Specific tests include:

  • Tests for each of the Base record types and device support. This should be based on ANJ's IOC Test Module branch so the tests can be written in Perl.
  • Any parts of libCom which don't currently have tests in libCom/test (C code)
  • Rewrite the tests from the epics-soft-test suite (C or Perl).

Convert makeBpt into Perl

The makeBpt tool in src/bpt generates non-linear break-point tables from data files. A fair bit of the C code is concerned with parsing the input file. There is no particular need for this code to be written in C, speed is not an issue, and eventually I hope to convert all programs needed for building EPICS into Perl code. Maybe start this project by writing a program to test the output, then convert the C code into Perl.

Support for 64-bit integer fields

ANJ has started adding support for DBF_INT64 and DBF_UINT64 types to the database, but the result has not been heavily tested and may not work on all target architectures (in particular VxWorks). See ANJ to work on this.

Other Tasks which could be done now

These items can be worked on immediately as they do not depend on any other work to have been completed.

Bug Triage and Fixing

See the Launchpad bug tracker. Bug triage involves understanding and replicating the bug and ensuring that it still exists in the latest release. Bugs tagged 'codeathon' are especially appropriate for fixing as Codeathon projects.

Review Merge Proposals

The Launchpad active review page lists branches which have been proposed for merging.

Documentation

Lots we can do here:

Application Developers Guide

  • Write a "Building and Porting EPICS" chapter
  • Restructure the AppDevGuide to make it more accessible
    • Combine the libCom and libCom/osi chapters
    • Maybe have a separate section for all callable APIs
    • Consider moving API documentation into the appropriate source files and using Doxygen or perldoc to extract it.

Other Documents

  • Look at all the documentation we have and see what's missing

VxWorks SMP Support

Recent versions of vxWorks (6.8+) support SMP but present a slightly different API that removes intLock() and taskLock(), and replaces taskVar with __thread variables. The current OSD routines for vxWorks will need significant modification to work with the new APIs. This will probably require creating a new OS-Class.

Can obviously only be done by someone with access to vxWorks 6.8 or later.

Extended Device Support

Experience converting device support layers to Extended support exposed the need to clean up the extended interface, i.e. reduce the code duplication which is currently required since the record knows nothing about the device change.

Libs affected: db, rec.

More flexible Access Security

The AS design defines each field's security level in the record description, and limits us to only levels 0 and 1. Some sites may want finer control over accesses: being able to put individual fields of a specific record into a different level, and having additional static levels (use a bitmask?) would help in a number of situations. It is not clear how much work this would entail though.

Priority: Low
Libs affected: dbStatic, db, as, ...

Authentication

From Ralph: Two or three institutes need a way to password-protect access to certain databases. The usual encryption libraries are freely available, but I guess this issue needs a lot more thinking. Granularity: by record, by connection = IOC, or centralised? Should the authentication last for the client's lifetime or should a short form (some security token or key) be sent with every CA data package?

Secure CA

Secure channel access protocol - add SSL wrappers around CA's TCP connections to make them secure.

Libs affected: ca, cas, rsrv

Things which can't be completed yet

These are tasks are waiting on other people or for other tasks to be completed before they can be finished, although there may be infrastructure to support them that can be worked on in parallel.

IOCSH Command Registration

Add DBD file syntax to convert DBD file information into a C header file which declares the command functions (similar to how we do record types in the current code-base). This header would then be included by the C file that defines those functions, thus the C compiler would do the work of ensuring that the routine implemented actually matches the command that will call it.

The command information in the DBD file will also be useful for future versions of the IRMIS parser for st.cmd scripts, where we need to be able to tie the device support configuration commands into IRMIS' need to map hardware devices to record addresses.

Libs affected: dbHost, all libs Registrar code.

Link Support

See original proposal by ANJ, which needs to be modified to add notify interfaces. This work is already under way, talk to ANJ.

Part of this development (June 2016) includes JSON Field Values.

Array Field Support

Make it easier to add array fields to record types without having to do as much coding in the record support. See Ben Franksen email and the work done by Nick Rees at the 2009 Codeathon.

Libs affected: dbHost, dbStatic, db, rec.

Variable Length Strings

The BaseString C++ APIs and classes were developed on the cvs-trunk branch and are being used by Jeff Hill in his replacement IOC server. This interface can form the basis of a new variable length string type for use in the database in addition to DBF_STRING.

Libs affected: dbHost, dbStatic, db, ...

Alternate DB File Formats

There have been calls to be able to replace the existing DB file format with an XML-based format. Currently the DB file parser is too highly integrated into the process of record creation to easily add a second format, but it would be feasible to separate these two and hence allow an expat-based or similar parser to be written. This would probably require a close analysis and upgrade to the locking arrangements of the database. Alternative to XML would be JSON, for which a parser is already being included in the jsonfm branch.

Libs affected: dbHost?, dbStatic.

New Record API

These requests all need a significant change to the record support API. They are also fairly closely related, so should be used as input into any new API that gets devised.

Asynchronous Gets (3.16++)

From Tim Mooney: It would be great if an input link could initiate processing, and get a value that is known to be the result of that processing, even if records to be processed are asynchronous. Currently, a record that needs completion-qualified data must know how to trigger the processing that generates that data.

From Ben Franksen: Implemented an asyncronous dbGetLinkCallback() routine. To do this properly though needs an upgrade to the record support API, but maybe could be shoe-horned into the the existing API by adding a dbGetLinkCallback() with a private callback pointer, and we'd require new record types to use this. What does this do to the device support interface though? Implications need study.

Andrew's Comment: This should be part of the implementation of linkSupport which I proposed in detail many years ago. The purpose of the "Compiling DBD Files" item above is to make extending the DBD file syntax easier to do in the future, which is a necessary part of that development.

Blocking Puts (3.16++)

From Tim Mooney: It would be nice to have a link attribute that could put, request processing, and wait for completion. We have this capability now, in dbCaPutCallback(), but it's selectable only at DCT time.

The idea is for a version of the SEQ record where rather than a fixed delay, the second link would wait for the completion callback from the first link. This should be possible today with a special record type, but Tim suggests it should be possible with any link field of any record type.

Andrew's Comment: This should be part of the implementation of linkSupport which I proposed in detail many years ago. The purpose of the "Compiling DBD Files" item above is to make extending the DBD file syntax easier to do in the future, which is a necessary part of that development.


Multiple Master problem (3.16++)

From Jeff: Changes to output values in a PLC or similar device initiated outside of the output record. This could be resolved adding a new routine to the RSET that the device support calls back when it has a new raw ouptut value to back-propagate to the VAL field. Currently done by some devices that set PACT and call the record's process() routine to use its second-half processing only.


Records Can't Block (3.16++)

From Tim Mooney: I think almost half the code in the sscan record is devoted to bookmarking. In the middle of some algorithm, the record needs to wait for external events; the only way back in is process(), and so the record has to have a complicated arrangement of switches that get it back into the algorithm it was executing, with enough state information to decide what to do next. At the same time, the record must check to see if it's now processing because of some out-of-band request from the user. Normal and exceptional processing must, therefore, be mixed together in the switch arrangement, and adding new behaviors to the record becomes a very hairy undertaking.

But what the record would really like to do is just wait with its context intact for events, and allow special() to release it from the wait if the user has requested something that requires this. I guess this means the record should include author-defined event types in the wait mask, and special() should be able to generate author-defined events.


Dynamically Loaded Modules (3.16++)

After converting the DBD load process into table registration, it should become easier to allow support modules (containing record types, device support and other drivers) to be discovered and loaded dynamically. Unloading is a very different matter though, we don't have the shutdown facilities to allow that.

Atomic Operations through CA (3.16++)

This is related to the Asynch Gets issue above; implement Read-Write (Swap, permits semaphores), Process-Read and Write-Process-Read (Command/Response) as atomic operations available through Channel Access.

Retrieved from "https://wiki-ext.aps.anl.gov/epics/index.php?title=Future_Development_Ideas&oldid=3813"