An important feature of DataForge framework is ability to work with meta-data. This documentation is automatically generated by grain script from markdown and html pieces. Such pieces are called shards. The shard consists of data in one of supported formats and meta-data header. The ordering of shards is automatically inferred from meta-data by the script.
The meta-data of each shards includes version of this precise shard last update date, it's label for reference and ordering information.
One of the main ideas of DataForge is that all changeable information is represented by Meta objects (called annotation in earlier versions). Meta is simple tree-like object that can conveniently store different data.
The naming of meta elements and values follows basic DataForge [navigation](naming and navigation) convention.
Meaning that elements and values could be called like child_name.grand_child_name.value_name
.
One can event use numbers as queries in such paths like child_name.grand_child_name[3].value_name
.
![]() |
The Meta object is a tree-like structure, which can contain other meta objects as branches (which are called elements) and Value objects as leafs. Both Values and Meta elements are organized in String-keyed maps. And each map element is a list of appropriate type. By requesting single Value or Meta element one is supposed to request first element of this list. Note that such lists are always immutable. Trying to change it may cause a error. While meta itself does not have any write methods and is considered to be immutable, some of its extensions do have methods that can change meta structure. One should be careful not to use mutable meta elements when one need immutable one. In order to conveniently edit meta, there is MetaBuilder class. |
The configuration is a very important extension of basic Meta class. It is basically a mutable meta which incorporates external observers. It is also important that while simple Meta knows its children knows its children, but could be attached freely to any ancestor, configuration has one designated ancestor that is notified than configuration is changed.
Note that putting elements or values to configuration follows the same naming convention as getting from it.
Meaning putting to some_name.something
will actually create or put to the node some_name
if it exists.
Otherwise, new node is created.
Context encapsulation.
![]() |
This section is under construction... |
Navigation
![]() |
This section is under construction... |
Data flow models
Actions
Tasks
The DataForge functionality is largely based on metadata exchange and therefore the main medium for messages between different parts of the system is Meta
object and its derivatives. But sometimes one needs not only to transfer metadata but some binary or object data as well.
In order to do so one should use an Envelope
format. It is a combined format for both text metadata and data in single block. An Envelope
container consists of three main components:
An envelope is a logical structure and its physical binary representation could be different for different purposes, but the default for envelope files or streams is the following:
Tag. First 30 bytes of file or stream is reserved for envelope properties binary representation:
#!
- two ASCII symbols, beginning of binary string.type
field: envelope format type and version. Depending on this value the rest of the binary string could be interpreted differently.metaType
field: metadata type and encoding.metaLength
field: metadata length in bytes including new lines and other separators.dataType
field: data format and type. This field is not necessary for some applications and could be used for other purposes.dataLength
field: the data length in bytes.!#
- two ASCII symbols, end of binary string.\r\n
- two bytes, new line.The values are read as binary and transformed into 4-byte unsigned tag codes.
Properties override. Properties could be overridden with text values using following notation:
#? <property key> : <property value>; <new line>
Any whitespaces before <property value>
begin are ignored. The ;
symbol is optional, but everything after it is ignored. Every property must be on a separate line. It is recommended to use universal new line sequence \r\n
to ensure correct work on any system.
Properties are accepted both in their textual representation or tag code.
Metadata block. Metadata in any accepted format. Additional formats could be provided by modules. The default metadata format is UTF-8 encoded XML (tag code 0x00000000). JSON format is provided by storage module.
One must note that metaLength
property is very important and in most cases is mandatory. It could be set to 0xffffffff
or -1
value in order to force envelope reader to derive meta length automatically, but different readers do it in a different ways, so it strongly not recommended to do it if data block is not empty.
Data block. Any other data. If dataLength
property is set to 0xffffffff
or -1
, then it is supposed that data block ends with the end of file or stream. Data block does not have any limitations for its content. It could even contain envelopes inside it!
The currently used envelope properties and codes are the following:
![]() |
This section is under construction... |
Storage plugin defines an interface between DataForge and different data storage systems such as databases, remote servers or other means to save and load data.
The main object in storage system is called Storage
. It represents a connection to some data storing back-end. In terms of SQL databases (which are not used by DataForge by default) it is equivalent of database. Storage
could provide different Loaders
. A Loader
governs direct data pushing and pulling. In terms of SQL it is equivalent of table.
Note: DataForge storage system is designed to be used with experimental data and therfore loaders optimized to put data online and then analyze it. Operations to modify existing data are not supported by basic loaders.
Storage system is hierarchical: each storage could have any number of child storages. So ot is basically a tree. Each child storage has a reference for its parent. The sotrage without a parent is called root storage. The system could support any number of root storages at a time using storage
context plugin.
By default DataForge storage module supports following loader types:
DataPoint
objects. PointLoader
contains information about DataPoint
DataFormat
. It is assumed that this format is just a minimum requirement for DataPoint
pushing, but implementation can just cut all fields tht are not contained in loader format.fragment
objects. Type of these objects is defined by generic and API does not define the format or procedure to read or right these objects.Loaders as well as Storages implement Responder
interface and and could accept requests in form of envelopes.
The FileStorage
is the default implementation of storage API.
![]() |
This section is under construction... |
DataForge control subsystem defines a general api for data acquisition processes. It could be used to issue commands to devices, read data and communicate with storage system or other devices.
The center of control API is a Device
class.
The device has following important features: