V4 Name Server - Revision history

KayKasemir: Reduced to essentials

2005-07-07T16:05:15Z

Reduced to essentials

← Older revision		Revision as of 16:05, 7 July 2005
Line 256:		Line 256:

	/// Connect to directory server		/// Connect to directory server
			/// Better: URL ala "LDAP://cn=Manager,dc=epics:secret@localhost:389"
			/// which would allow other services without changing the API?
	bool connect(const char *host = "localhost:389",		bool connect(const char *host = "localhost:389",
	const char *user = "cn=Manager,dc=epics",		const char *user = "cn=Manager,dc=epics",
Line 262:		Line 264:
	/// Disconnect from directory server		/// Disconnect from directory server
	void disconnect();		void disconnect();

	~~/// Get some info string~~
	~~void api_info(char *buf, size_t len);~~

	~~/// Useless API~~
	~~void dump_PVs();~~

	/// Directory entry for one PV		/// Directory entry for one PV

KayKasemir: Added test API

2005-07-07T16:02:23Z

Added test API

← Older revision		Revision as of 16:02, 7 July 2005
Line 238:		Line 238:
	Elapsed Per Item = 0.00001683 sec, 59427.3 Items per sec, 31.4 Mbps		Elapsed Per Item = 0.00001683 sec, 59427.3 Items per sec, 31.4 Mbps
	Search tries per chan - mean=1.000930 std dev=0.030482 min=1.000000 max=2.000000		Search tries per chan - mean=1.000930 std dev=0.030482 min=1.000000 max=2.000000

			== API ==
			[[V4 CA Interfaces]] already has a lookup interface for a name service
			that uses asynchronous callbacks and a property catalog to return the results.

			This is the interface I used for the test code. It is synchronous without configurable timeout (bad), but includes addition and removal of names (good), and the PVEntry lists the actual properties of an entry.

			/// Client-Interface to the EPICS directory
			class Directory
			{
			public:
			/// Constuctor
			Directory();

			/// Destructor (will disconnect)
			~Directory();

			/// Connect to directory server
			bool connect(const char *host = "localhost:389",
			const char *user = "cn=Manager,dc=epics",
			const char *password = "secret");

			/// Disconnect from directory server
			void disconnect();

			/// Get some info string
			void api_info(char *buf, size_t len);

			/// Useless API
			void dump_PVs();

			/// Directory entry for one PV

			/// Unclear how to best deal with this with regard to real-time OS
			/// where we don't want to allocate too much mem at runtime.
			/// This API allows caller to reuse the same 'PVEntry' over and over.
			static const size_t SLEN = 100;
			static const size_t MAXCAS = 3;
			static const char PATH_SEP = '/';
			class PVEntry
			{
			public:
			char pv[SLEN];
			char path[SLEN];
			char description[SLEN];
			char cas[MAXCAS][SLEN];
			};

			bool Directory::add_pventry(const PVEntry *pv);

			/// Given pv->pv, fills remaining PVEntry information from directory
			bool get_pventry(PVEntry *pv);

			bool delete_pventry(const char pv, const char path);
			};

	=== To be investigated ===		=== To be investigated ===

JeffHill: /* JOH: For comparison purposes, here is the performance of CA's built-in name resolution. */

2005-07-06T21:34:31Z

JOH: For comparison purposes, here is the performance of CA's built-in name resolution.

← Older revision		Revision as of 21:34, 6 July 2005
Line 234:		Line 234:
	==== JOH: For comparison purposes, here is the performance of CA's built-in name resolution. ====		==== JOH: For comparison purposes, here is the performance of CA's built-in name resolution. ====

	Testing with ~~10000~~ channels named joh:bill		Testing with 100000 channels named joh:bill
	channel connect test		channel connect test
	Elapsed Per Item = 0.~~00002349~~ sec, ~~42570~~.2 Items per sec, 22.5 Mbps		Elapsed Per Item = 0.00001683 sec, 59427.3 Items per sec, 31.4 Mbps
	Search tries per chan - mean=1.~~006200~~ std dev=0.~~078496~~ min=1.000000 max=2.000000		Search tries per chan - mean=1.000930 std dev=0.030482 min=1.000000 max=2.000000

	=== To be investigated ===		=== To be investigated ===

JeffHill at 20:12, 6 July 2005

2005-07-06T20:12:10Z

← Older revision		Revision as of 20:12, 6 July 2005
Line 232:		Line 232:
	have to request data for specific PV names, not by pattern.		have to request data for specific PV names, not by pattern.

	JOH: For comparison purposes, here is the performance of CA's built-in name resolution.		==== JOH: For comparison purposes, here is the performance of CA's built-in name resolution. ====

	Testing with 10000 channels named joh:bill		Testing with 10000 channels named joh:bill
			channel connect test
	channel connect test		Elapsed Per Item = 0.00002349 sec, 42570.2 Items per sec, 22.5 Mbps
			Search tries per chan - mean=1.006200 std dev=0.078496 min=1.000000 max=2.000000
	Elapsed Per Item = 0.00002349 sec, 42570.2 Items per sec, 22.5 Mbps

	Search tries per chan - mean=1.006200 std dev=0.078496 min=1.000000 max=2.000000

	=== To be investigated ===		=== To be investigated ===
Line 248:		Line 245:
	* API: What type of API would EPICS tools use? Whatever LDAP library they use? An 'EPICS wrapper' around LDAP? Gasper suggested to look at the Java JNDI API for inspiration.		* API: What type of API would EPICS tools use? Whatever LDAP library they use? An 'EPICS wrapper' around LDAP? Gasper suggested to look at the Java JNDI API for inspiration.
	* Common Database headaches:		* Common Database headaches:
	** To simply get a PV into LDAP, one has to check if this PV already exists and then either 'add' or 'modify'.		** To simply get a PV into LDAP, one has to check if this PV already exists and then either 'add' or 'modify'.
	** What if a channel is no longer available? An IOC shutting down could remove PVs from the LDAP server, but an IOC that crashes won't.		** What if a channel is no longer available? An IOC shutting down could remove PVs from the LDAP server, but an IOC that crashes won't.
	* Replication: How to use it, how fast is it etc.		* Replication: How to use it, how fast is it etc.
	* Authentication and encryption: How hard is it to configure?		* Authentication and encryption: How hard is it to configure?

JeffHill at 20:07, 6 July 2005

2005-07-06T20:07:08Z

← Older revision		Revision as of 20:07, 6 July 2005
Line 231:		Line 231:
	PVs that match a pattern. Unfortunately, typical CA clients will		PVs that match a pattern. Unfortunately, typical CA clients will
	have to request data for specific PV names, not by pattern.		have to request data for specific PV names, not by pattern.

			JOH: For comparison purposes, here is the performance of CA's built-in name resolution.

			Testing with 10000 channels named joh:bill

			channel connect test

			Elapsed Per Item = 0.00002349 sec, 42570.2 Items per sec, 22.5 Mbps

			Search tries per chan - mean=1.006200 std dev=0.078496 min=1.000000 max=2.000000

	=== To be investigated ===		=== To be investigated ===

KayKasemir: views

2005-06-29T16:17:48Z

views

← Older revision		Revision as of 16:17, 29 June 2005
Line 44:		Line 44:

	Whenever a tool like a command-line 'caget' or the 'probe' GUI requires a PV name,		Whenever a tool like a command-line 'caget' or the 'probe' GUI requires a PV name,
	~~the convention might~~ be ~~that "fred" will address all~~ properties ~~of the channel~~ "fred".		it needs to be decided what properties "fred" would access. In the past, it was equal
	In ~~order to get~~ the ~~value~~, ~~one would have~~ to ~~use~~ "fred.~~value~~".		to "fred.VAL". There is much more detail in the 'view' section of the
	~~Note that~~ the ~~"fred" == "fred.VAL" default would no longer apply.~~		[[V4 DBD Statement Syntax]].
	~~This change would scale better when accessing hierarchical properties:~~
	~~"fred.limits" would get all limits, while "fred.limits.display.high" would only get~~
	~~the requested element~~.

	== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==		== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==

KayKasemir: defaults, properties

2005-06-27T19:33:32Z

defaults, properties

← Older revision		Revision as of 19:33, 27 June 2005
Line 24:		Line 24:
	* PV meta information like engineer-in-charge, CA security rules, ...		* PV meta information like engineer-in-charge, CA security rules, ...

	== Channels, Records, PVs ==		== Channels, Records, PVs, Properties ==
	We currently have records with fields and channels with properties.		We currently have records with fields and channels with properties.
	Each field of a record can become a channel name,		Each field of a record can become a channel name,
	the field itself ends up in the 'value' property of the channel,		the field itself ends up in the 'value' property of the channel,
	and the record decides how fill the remaining properties.		and the record decides how to fill the remaining properties.

	Will the directory list all record names or all possible channels?		Will the directory list all record names or all possible channels?
Line 38:		Line 38:
	I think for now that only listing record names is enough in order to allow		I think for now that only listing record names is enough in order to allow
	connections.		connections.

			The CA client would then have to query the CA server for all the available properties
			of a PV, and note that properties might even change at runtime when we allow
			online add & delete.

			Whenever a tool like a command-line 'caget' or the 'probe' GUI requires a PV name,
			the convention might be that "fred" will address all properties of the channel "fred".
			In order to get the value, one would have to use "fred.value".
			Note that the "fred" == "fred.VAL" default would no longer apply.
			This change would scale better when accessing hierarchical properties:
			"fred.limits" would get all limits, while "fred.limits.display.high" would only get
			the requested element.

	== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==		== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==

KayKasemir: channels or records?

2005-06-23T23:08:05Z

channels or records?

← Older revision		Revision as of 23:08, 23 June 2005
Line 16:		Line 16:

	As Larry Hoff pointed out, we really want a ''Directory'' server for		As Larry Hoff pointed out, we really want a ''Directory'' server for
	EPICS, something beyond a name ~~serve~~. A directory will not only map PV		EPICS, something beyond a name server. A directory will not only map PV
	names to IP addresses of ChannelAccess servers, but also support		names to IP addresses of ChannelAccess servers, but also support
	* wildcard searches, reports on number of channels per IOC etc.		* wildcard searches, reports on number of channels per IOC etc.
Line 23:		Line 23:
	* backup/restore locations for a PV		* backup/restore locations for a PV
	* PV meta information like engineer-in-charge, CA security rules, ...		* PV meta information like engineer-in-charge, CA security rules, ...

			== Channels, Records, PVs ==
			We currently have records with fields and channels with properties.
			Each field of a record can become a channel name,
			the field itself ends up in the 'value' property of the channel,
			and the record decides how fill the remaining properties.

			Will the directory list all record names or all possible channels?
			I think only record names.

			Will it include the record type? Not enough information if we add user-defined fields.
			Will it lists all fields of a record?

			I think for now that only listing record names is enough in order to allow
			connections.

	== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==		== LDAP, Directory, RDB, Schema, Entries, dn, cn, dc, ... ==

KayKasemir: C numbers for add, locate, delete

2005-06-09T15:58:29Z

C numbers for add, locate, delete

← Older revision		Revision as of 15:58, 9 June 2005
Line 124:		Line 124:
	ldapsearch -x -LLL -E pr=10/noprompt -b 'dc=epics' '(cn=testpv*)' cas		ldapsearch -x -LLL -E pr=10/noprompt -b 'dc=epics' '(cn=testpv*)' cas

	=== Performance ===		== Performance ==

	Based on the default OpenLDAP server config,		Based on the default OpenLDAP server config with an additional
			index "cn pres,eq" on the common name of the PVs,
	PVs with names 'testrec0000000001', 'testrec0000000002' etc.		PVs with names 'testrec0000000001', 'testrec0000000002' etc.
	were added, then looked up one-by-one via their exact name,		were added, then looked up one-by-one via their exact name,
	~~then via a pattern 'testrec*',~~ and finially deleted one by one.		and finially deleted one by one.

			When using a C++ test client based on the OpenLDAP library,
			the results for smaller numbers are:
			Add 10000 records : 22.76 secs
			Locate 10000 records : 6.49 secs
			Delete 10000 records : 22.20 secs

	~~The following is for a perl-script based~~ test,		In another test, a very rough estimated PV count of the SNS linac
	~~using a pure perl LDAP client library,~~		was used:
	~~because that was the easiest to use in order to get~~ a ~~first~~		Add 200000 records : 675.89 secs (38.74 CPU)
	~~understanding~~ of the ~~whole elephant.~~		Locate 200000 records : 142.85 secs (27.91 CPU)
	~~A C test based on the OpenLDAP client library turned~~		Delete 200000 records : 807.28 secs (37.28 CPU)
	~~out to be MUCH faster~~:
	~~Locating 10000~~ records ~~from C,~~
	~~retrieving the 'cas' attribute for each without printing it~~: 6.~~2 seconds~~

	In ~~the first~~ test, the individual PV search actually used an LDIF		Those 200000 records use about 900MB of file space.
	filter with exact match like 'testrec0000000001':		The results indicate that one can add 300 .. 440 records/sec,
			locate 1400.. 1540 records/sec and delete 250..450 records/sec.
			Without an index, the addition and deletion will be faster
			at the expense of lookup times.

			A typical SNS LLRF IOCs has 2300 records.
			On startup, sending those to the directory server would take around 7 seconds.
			A typical SNS LLRF overview screen has about 400 PVs,
			and current connection times vary from less than a second to
			several tens of seconds, depending on how well the search requests
			reach the IOCs.
			Resolving their CA servers would take about 0.3 seconds with LDAP,
			which would then be followed by the time required to actually connect
			to those servers.

			In previous test using perl, the individual PV search actually used an LDIF
			filter with exact match like 'testrec0000000001'. In addition, all PVs matching
			a filter 'testrec*' were retrieved, and no index was used:
	Adding 1000 records: 4 wallclock secs ( 2.29 usr + 0.14 sys = 2.43 CPU)		Adding 1000 records: 4 wallclock secs ( 2.29 usr + 0.14 sys = 2.43 CPU)
	Locating 1000 records: 49 wallclock secs ( 4.10 usr + 0.21 sys = 4.31 CPU)		Locating 1000 records: 49 wallclock secs ( 4.10 usr + 0.21 sys = 4.31 CPU)
Line 168:		Line 188:

	Adding 200000 records: 1293 wallclock secs (483.65 usr + 33.20 sys = 516.85 CPU)		Adding 200000 records: 1293 wallclock secs (483.65 usr + 33.20 sys = 516.85 CPU)

	~~Those 200000 records created a little over 1GB of BerkeleyDB files.~~

	"index cn,dc pres,eq,approx,sub" takes longer to insert/delete		"index cn,dc pres,eq,approx,sub" takes longer to insert/delete
	without improved search performance:		without improved search performance:
	Adding 10000 records: 104 wallclock secs (26.25 usr + 1.67 sys = 27.92 CPU)		Adding 10000 records: 104 wallclock secs (26.25 usr + 1.67 sys = 27.92 CPU)
	Locating 10000 records: 45 wallclock secs (32.86 usr + 1.55 sys = 34.41 CPU)		Locating 10000 records: 45 wallclock secs (32.86 usr + 1.55 sys = 34.41 CPU)
	Match all 10000: 15 wallclock secs (12.31 usr + 0.47 sys = 12.78 CPU)		Match all 10000: 15 wallclock secs (12.31 usr + 0.47 sys = 12.78 CPU)
	Delete 10000 records: 97 wallclock secs (17.89 usr + 1.34 sys = 19.23 CPU)		Delete 10000 records: 97 wallclock secs (17.89 usr + 1.34 sys = 19.23 CPU)
Line 188:		Line 206:
	Searching PVs one PV name at a time is slower than requesting all		Searching PVs one PV name at a time is slower than requesting all
	PVs that match a pattern. Unfortunately, typical CA clients will		PVs that match a pattern. Unfortunately, typical CA clients will
	have to request data for specific PV names, not by pattern,		have to request data for specific PV names, not by pattern.
	~~so the performance is below the ideal ChannelAccess case where search requests~~
	~~go out onto an idle network.~~

	~~Some interpolations of the benchmark:~~
	~~A typical SNS LLRF IOCs has 2300 records.~~
	~~On startup, sending those to the directory server would take 10 seconds.~~

	~~A typical SNS LLRF overview screen has about 400 PVs,~~
	~~and current connection times vary from less than a second to~~
	~~several tens of seconds, depending on how well the search requests~~
	~~reach the IOCs.~~
	~~Resolving their CA servers would take 2 seconds with LDAP,~~
	~~which would then be followed by the time required to actually connect~~
	~~to those servers~~.

	=== To be investigated ===		=== To be investigated ===

KayKasemir: first C test result

2005-05-31T21:52:01Z

first C test result

← Older revision		Revision as of 21:52, 31 May 2005
Line 130:		Line 130:
	were added, then looked up one-by-one via their exact name,		were added, then looked up one-by-one via their exact name,
	then via a pattern 'testrec*', and finially deleted one by one.		then via a pattern 'testrec*', and finially deleted one by one.


			The following is for a perl-script based test,
			using a pure perl LDAP client library,
			because that was the easiest to use in order to get a first
			understanding of the whole elephant.
			A C test based on the OpenLDAP client library turned
			out to be MUCH faster:
			Locating 10000 records from C,
			retrieving the 'cas' attribute for each without printing it: 6.2 seconds

	In the first test, the individual PV search actually used an LDIF		In the first test, the individual PV search actually used an LDIF