Datasets
Last updated
Last updated
With the introduction of Datasets, it is now possible for large amounts of user, hardware, network and other supporting data to be sent to and received from remote systems. While this has been possible for some time using Pollers, they are not specific to handling large amounts of data in the way that Datasets are.
Datasets provide easy configuration that is supported by platform intelligence and automation, making it is incredibly easy to setup a robust and efficient mechanism for handling large sets of data.
Datasets require the latest Unifi Global Utility for them to build the necessary configuration correctly.
Each Dataset will create the following configuration:
The Send message is used for sending the data in an attachment to the other system. The name is automatically generating using the prefix "Send" and the name of the table.
The Process message is used for processing the record data both inbound and outbound. The name is automatically generating using the prefix "Process" and the name of the table.
The Import set table is used for staging inbound data. The name is automatically generated using the prefix "dataset" with the sys_id of the dataset.
The Scheduled import record is automatically created for processing import data. It is executed when an inbound Dataset Request has created the import set rows and is ready to be processed.
The Transform map and associated coalesce fields are used for processing the import set.
Data is automatically collected, transformed, packaged and sent to another system on a schedule. Each export creates one or more Dataset Requests depending on the number of records and a series of limits configurable on the Dataset.
Unifi automatically creates a Process message which can be used with outbound Fields and Field maps (via the Source to Stage and Stage to Request Message Scripts) to extract the data from the specified records.
The extracted data is written to an attachment which is then sent using the Send message which is also automatically created by Unifi.
When an inbound Send message (as specificed on the Dataset) with a data attachment is received, the attachment will be processed with each record being inserted into an import set table directly related to the Dataset. The import set table is automatically created and maintained during the Integration Build process.
Unifi uses ServiceNow Import Sets since they offer a well understood mechanism for importing data with performance benefits. The mapping is automatically managed by Unifi and handled through a transform script which uses inbound Fields and Field maps (via the Stage to Target Message Script) to transform the data and write it to the target records.
The process message related to the Dataset is used for inbound and outbound mapping and transformation.
Simply create the fields that you want to export/import and configure them in the same way you would for any other Unifi message. Note: the field maps will likely need to be specific to the dataset fields.
Use the Coalesce field (specific to Dataset messages) to indicate which field should be used for identifying existing records to update during inbound processing.
