8.8. Generating XML output from the OPC DataHub

OPC DataHub™ : Version 6.4
	Chapter 8. OPC to Web

The OPC DataHub produces XML-formatted data to web-based clients in a number of ways. The choice of mechanism will be determined by the needs of the client application. Generally, these mechanisms can be categorized into either streamed or polled methods.

Streaming XML Data

The OPC DataHub Web Server provides an efficient method for streaming data over a TCP/IP socket. The initial socket connection is negotiated using HTTP/HTTPS. Once the socket is established, a separate thread in the OPC DataHub takes over responsibility for the socket. After the initial HTTP negotiation, all communication is uni-directional. That is, the DataHub emits data to the client application, but expects no data to be transmitted by the client. The XML format can be pre-defined or user-configured. The actual formatting of the data is performed via scripts running in the DataHub scripting language, Gamma.

The streamed XML mechanism makes use of the same underlying technology that provides the "Streaming AJAX" in the OPC DataHub. Only the data format is different.

Polling XML Data

The OPC DataHub Web Server provides two methods for receiving XML data via polling. In the common case, the OPC DataHub offers a special URL that directly accesses the data set within the DataHub engine to construct an XML string. The XML format is pre-defined. The string construction is performed entirely in memory, so this method is very efficient.

If the client requires a different XML format, this can be provided via an ASP page. The DataHub uses Gamma as its ASP language, meaning that a request for a web page may trigger Gamma scripting calls that produce the XML data in any format that the web developer requires.

8.8.1. Streaming XML How-To

8.8.1.1. Built-in Streaming Data

The built-in streaming data is accessed using a URL of the form: http://hostname:port/stream?arguments. The arguments are separated by the & symbol, and can be any combination of:

name=pointname1|pointname2|...

This is a list of individual point names that will be retrieved from the DataHub. The point names must be fully-qualified, such as DataSim:Sine rather than just Sine. The point names are separated by the pipe character, "|". If the name argument is omitted then the domain argument must be provided. Example:

name=DataSim:Sine|DataSim:Ramp|DataSim:Square

template=[json,jsonp,djson,djsonp,xml,lisp]

This selects the output formatting from the available options shown in the list. The template will determine the default head, tail, prefix and suffix. Example:

template=xml

head=file name

The head is the name of a file relative to the document root of the DataHub Web Server that will be transmitted once when the connection is first made. This can contain information like XML version or web page header information. Example:

head=my_header_file

tail=file name

The tail is the name of a file relative to the document root of the DataHub Web Server that will be transmitted immediately prior to terminating the connection. The connection will be terminated automatically by the DataHub when the msglimit is reached. If msglimit is 0, this file will never be transmitted.

prefix=<any string>

The prefix XML tag is transmitted prior to every data update. The DataHub will transmit data as it becomes available, subject to the throttle setting. If there is more than one data value to be transmitted, the prefix is transmitted once for every group of data points, not once per data point. Example:

prefix=<points>

suffix=</any string>

The suffix XML tag is transmitted after every data update. See prefix above for more details. Example:

suffix=</points>

msglimit=a non-negative integer

Some clients need to periodically close and re-open a streaming connection in order to clear resources accumulated while the connection is open. This will allow the client to indicate how many data updates can be transmitted before the DataHub must close the connection. It is the client's responsibility to re-open the connection once it is closed. If this value is 0, the OPC DataHub will never intentionally close the connection. The DataHub will only transmit the tail file before closing the connection, so if this value is 0, the tail file will never be transmitted. Example:

msglimit=10000

throttle=a non-negative floating-point number

The number of seconds to wait before sending data, allowing a client to request a maximum update rate from the OPC DataHub. That is, if data is changing faster than throttle seconds then the DataHub will accumulate data points and transmit them only after throttle seconds have passed since the previous transmission. If a data point changes more than once within this period, only the most recent value is transmitted. Set throttle to 0 to indicate that the OPC DataHub should transmit all data without delay. Example:

throttle=0.25

user=a non-empty string

If the OPC DataHub security settings require a user name and password for a TCP connection, the client can supply those here. Example:

user=my_name&pass=my_pass

pass=a non-empty string

See above.

domain=a domain name list

The client may request all of the data in a data domain instead of naming data points individually. If the domain argument is provided, the name argument should not be provided. Multiple domain names can be separated by the pipe character, "|". Example:

domain=DataPid

Due to a bug in the OPC DataHub, this argument is not currently functional. Instead, use: name=domain_name&children=1&recursive=1. Example:

name=DataPid&children=1&recursive=1

children=[0,1]

When the client supplies a name argument, it can also supply the children argument to indicate whether to also retrieve any child points of that point. If multiple points are supplied in the name argument, then the children argument will apply to all points. A value of 0 indicates not to retrieve children. A value of 1 indicates to retrieve children. This is further affected by the recursive setting below. Example:

children=1

recursive=[0,1]

If the children argument is 1, then the value of recursive will determine whether to walk the data hierarchy starting at the named point, retrieving all descendants of that point. If the value of recursive is 0, then only the direct children of each point will be retrieved. If the value of recursive is 1, then all descendants of the named points will be retrieved. Example:

recursive=1

Example

To retrieve all of the points in the DataPid domain, at a maximum update rate of 5 Hz, for a maximum of 10000 data changes, the URL would be:

http://localhost/stream?name=DataPid&children=1&recursive=1&msglimit=30&throttle=0.2&template=xml

The resulting output would look like this:

<points>
<point name="DataPid:PID1.Controller.Kd" value="0.01" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Controller.Ki" value="0.5" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Controller.Kp" value="0.25" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Mv" value="37.2139761632173" type="1" quality="192"
timestamp="1275683572.9200001" />
<point name="DataPid:PID1.Plant.Ki" value="0.5" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Plant.Kp" value="2" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Pv" value="55.551238388592" type="1" quality="192"
timestamp="1275683572.9200001" />
<point name="DataPid:PID1.Range.Amplitude" value="100" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Range.Offset" value="50" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Setpoint.AutoMode" value="1" type="2" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Setpoint.AutoTime" value="5" type="1" quality="192"
timestamp="1275682823.9979999" />
<point name="DataPid:PID1.Setpoint.SpInput" value="0" type="1" quality="192"
timestamp="-2147483648" />
<point name="DataPid:PID1.Sp" value="72.4715414899136" type="1" quality="192"
timestamp="1275683572.1540003" />
<point name="DataPid:PID1.UpdateFrequency" value="10" type="1" quality="192"
timestamp="1275682823.9979999" />
</points>
<points>
<point name="DataPid:PID1.Mv" value="38.3540008662675" type="1" quality="192"
timestamp="1275683573.1379995" />
<point name="DataPid:PID1.Pv" value="57.5623437038486" type="1" quality="192"
timestamp="1275683573.1379995" />
</points>
<points>
<point name="DataPid:PID1.Mv" value="39.2907100165968" type="1" quality="192"
timestamp="1275683573.3569999" />
<point name="DataPid:PID1.Pv" value="59.5695627360927" type="1" quality="192"
timestamp="1275683573.3569999" />
</points>
<points>
<point name="DataPid:PID1.Mv" value="40.0353526249595" type="1" quality="192"
timestamp="1275683573.5760002" />
<point name="DataPid:PID1.Pv" value="61.5352593843648" type="1" quality="192"
timestamp="1275683573.5760002" />
</points>
<points>
<point name="DataPid:PID1.Mv" value="40.6012237044704" type="1" quality="192"
timestamp="1275683573.7950006" />
<point name="DataPid:PID1.Pv" value="63.4279691691699" type="1" quality="192"
timestamp="1275683573.7950006" />
</points>

8.8.1.2. Customizing the Built-in Streaming Data

The format of streaming web data is controlled through a Gamma script provided in require\WebstreamSupport.g in the DataHub installation directory. You can add your own format templates to allow you to specify how the data is presented to your program. For example, the default XML layout looks like this:

<points>
<point name="DataSim:Sine" value="0.5" type="1" quality="192" timestamp="1275683573.7950006" />
</points>

There is no head or tail string. The prefix string is <points> and the suffix string is </points>. In addition there are three available format strings to determine how to format different data types:

`stringformat`	- a format string that will be used to prepare string data
`numberformat`	- a format string that will be used to prepare numeric (real and integer) data
`arrayformat`	- a format string that will be used to prepare array data

In the default XML template, these are:

`stringformat`	- `<point name=%s value=%w type="%d" quality="%d" timestamp="%g" />`
`numberformat`	- `<point name=%s value=%s type="%d" quality="%d" timestamp="%g" />`
`arrayformat`	- `<point name=%s value=%s type="%d" quality="%d" timestamp="%g" />`
`separator`	- a string used to separate multiple points between a single set of prefix and suffix strings. E.g., `","`

Each of the format strings must encode 5 input parameters, supplied in order as:

`name`	- a string representing the full point name
`value`	- the value of the point. This will be one of string, number or array
`type`	- a number representing the data type, where `0` = string or array, `1` = floating point number, `2` = integer number
`quality`	- an OPC quality as an integer number
`timestamp`	- a UNIX time stamp as the number of seconds since January 1, 1970 UTC

Arrays are encoded as strings of the form "['value1','value2','value3',...]".

These parameters are specified by modifying the definition of the Gamma class WebstreamSupport. The original definition is found in the file require\WebstreamSupport.g. It is wise not to modify this file, but instead to add your modifications in a separate file as follows:

Create your own Gamma file, e.g., MyCustomWebstream.g.
Remove all existing code in this file

Insert the following code:

require ("WebstreamSupport");

// Ensure that there is an instance variable for my new template
if (!has_ivar(WebstreamSupport, #my_custom))
{
    class_add_ivar(WebstreamSupport, #my_custom);

    // Re-instantiate the helper instance with the new instance variable included
    Webstream = ApplicationSingleton (WebstreamSupport);
}
Webstream.my_custom = new WebstreamTemplate ("CUSTOM", "", "my_custom",
                                             "<data>\n",
                                             "</data>\n",
                                             0, 0.0,
                                             "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
                                             "",
                                             "<point name=%s value=%w type=\"%d\" quality=\ %d\" timestamp=\"%g\" />\n",
                                             "<point name=%s value=%s type=\"%d\" quality=\"%d\" timestamp=\"%g\" />\n",
                                             //"<point name=%s value=%s type=\"%d\" quality=\"%d\" timestamp=\"%g\" />\n",
                                             " ");

Modify the code to match your requirements.

The code creates a new template member in the class WebstreamSupport and then assigns it a set of parameters by creating a WebstreamTemplate. The constructor to WebstreamTemplate takes the following arguments:

`tplname`	- an arbitrary string. It is not used.
`pointnames`	- a string of pipe-separated default point names, should be ""
`template_id`	- the name of the template supplied in the /stream URL
`prefix`	- a string that is prefixed to every group of data changes
`suffix`	- a string that is suffixed to every group of data changes
`msglimit`	- default msglimit as above
`throttle`	- default throttle as above
`head`	- a head string (not a file name as above) that is transmitted once at connection
`tail`	- a tail string that is transmitted immediately prior to the DataHub disconnecting
`stringformat`	- as above
`numberformat`	- as above
`arrayformat`	- as above. Depending on your version of OPC DataHub, this argument may be missing.
`separator`	- as above

Ensure that your new script runs when the DataHub starts.
Run your script. You should now be able to query data from a URL like:

http://localhost/stream?name=DataPid&children=1&recursive=1&template=my_custom

8.8.2. Polling XML How-To

Typical AJAX applications retrieve data through polling, using a Javascript function called XmlHttpRequest. This function makes an asynchronous HTTP request to a URL that returns an XML document. The Javascript application parses this XML document to extract relevant information.

8.8.2.1. Built-in HTTP Data Polling

The OPC DataHub implements a built-in XML polling mechanism that constructs the XML document entirely in memory to minimize CPU usage. The format of this document is not user-configurable, and is similar to the output produced by the streaming data facility when specifying template=xml.

The built-in streaming data is accessed using a URL of the form: http://hostname:port/points?arguments. The arguments are separated by the & symbol, and can be any combination of:

name=pointname1|pointname2|...

name=DataSim:Sine|DataSim:Ramp|DataSim:Square

domain=a domain name list

domain=DataPid

Due to a bug in the OPC DataHub, this argument is not currently functional. Instead, use: name=domain_name&children=1&recursive=1. Example:

name=DataPid&children=1&recursive=1

style=[json,jsonp,djson,djsonp,xml,lisp]

This selects the output formatting from the available options shown in the list. Example:

style=xml

prefix=<any string>

prefix=<points>

suffix=</any string>

The suffix XML tag is transmitted after every data update. See prefix above for more details. Example:

suffix=</points>

8.8.2.2. Customized HTTP Data Polling

Since the built-in data polling is not user-configurable, it is not an appropriate mechanism for generating custom data formats. Instead, customized XML data can be emitted using a user-supplied ASP page. The ASP mechanism within the OPC DataHub Web Server uses the Gamma scripting language as its ASP language. This means that an ASP page has access to the live data in the DataHub and any Gamma functions such as database calls. The Gamma calls can be used to generate strings that are inserted into the page.

A simple ASP page that generates an XML may look like this:

<?xml version="1.0" encoding="UTF-8"?>
<points>
    <point name="DataSim:Sine" value="<%= $DataSim:Sine %>" />
</points>

In this example, the structure of the document is simple XML, with data from Gamma calls embedded within <%= %> delimiters. The result of evaluating the Gamma expression is converted to a string and then inserted into the file, replacing the <%= %> delimiters and all text between those delimiters. The Gamma script between these delimiters must be a single Gamma expression, not a statement. For example, the expression 2+2 is legal, but the statement a=2+2; is not.

A more complex example could implement a loop in Gamma to query the data set in the DataHub and provide a value for every data point.

<?xml version="1.0" encoding="UTF-8"?>
<points>
<%
local points = datahub_points("DataPid", nil);
with point in points do
{
    if ((point.flags & 0x30) == 0) // ensure point is not an assembly
    {
        %>
        <point name="<%= point.name %>" value="<%= point.value %>" />
        <%
    }
}
%>
</points>

In this example, the delimiters <% %> are used to indicate Gamma script code that will not generate a replacement string. The Gamma script between <% and %> must be one or more Gamma statements, not expressions. Notice that Gamma statements may be broken up across occurrences of the <% %> delimiters, allowing a natural mechanism for specifying the XML portion of the file.

The ASP file can be placed in any subdirectory of the document root of the DataHub Web Server. The client application can then simply make repeated reads of this file to retrieve the current data values.


8.7. Creating Passwords		Chapter 9. OPC to Excel