# Datasets

With the introduction of Datasets, it is now possible for large amounts of user, hardware, network and other supporting data to be sent to and received from remote systems. While this has been possible for some time using Pollers, they are not specific to handling large amounts of data in the way that Datasets are.

Datasets provide easy configuration that is supported by platform intelligence and automation, making it is incredibly easy to setup a robust and efficient mechanism for handling large sets of data.

<figure><img src="https://2763208740-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZR0dkdEH0Mm7amiIzsdG%2Fuploads%2FHFHTkBorRikGBAdca1W1%2Fimage.png?alt=media&#x26;token=f2562215-053c-4ffe-8b7f-b23fc9c05350" alt=""><figcaption><p>Unifi Dataset</p></figcaption></figure>

## Integration Build

{% hint style="info" %}
Datasets require the latest [Unifi Global Utility](https://docs.sharelogic.com/unifi/4.0/install/installation/global-utility) for them to build the necessary configuration correctly.
{% endhint %}

Each Dataset will create the following configuration:

* The **Send message** is used for sending the data in an attachment to the other system. The name is automatically generating using the prefix "Send" and the name of the table.
* The **Process message** is used for processing the record data both inbound and outbound. The name is automatically generating using the prefix "Process" and the name of the table.
* The **Import set table** is used for staging inbound data. The name is automatically generated using the prefix "dataset" with the sys\_id of the dataset.
* The **Scheduled import record** is automatically created for processing import data. It is executed when an inbound Dataset Request has created the import set rows and is ready to be processed.
* The **Transform map** and associated coalesce fields are used for processing the import set.

<figure><img src="https://2763208740-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZR0dkdEH0Mm7amiIzsdG%2Fuploads%2Fu5IV2ExKeO3jUnENgaNx%2Fimage.png?alt=media&#x26;token=24b5ad66-ede0-46fd-863c-81093ec7952a" alt=""><figcaption><p>Dataset messages which have been automatically created</p></figcaption></figure>

## Outbound Data

Data is automatically collected, transformed, packaged and sent to another system on a schedule. Each export creates one or more Dataset Requests depending on the number of records and a series of limits configurable on the Dataset.

Unifi automatically creates a Process message which can be used with outbound Fields and Field maps (via the Source to Stage and Stage to Request Message Scripts) to extract the data from the specified records.

The extracted data is written to an attachment which is then sent using the Send message which is also automatically created by Unifi.

## Inbound Data

When an inbound Send message (as specificed on the Dataset) with a data attachment is received, the attachment will be processed with each record being inserted into an import set table directly related to the Dataset. The import set table is automatically created and maintained during the Integration Build process.

Unifi uses ServiceNow Import Sets since they offer a well understood mechanism for importing data with performance benefits. The mapping is automatically managed by Unifi and handled through a transform script which uses inbound Fields and Field maps (via the Stage to Target Message Script) to transform the data and write it to the target records.

## Fields

The process message related to the Dataset is used for inbound and outbound mapping and transformation.

Simply create the fields that you want to export/import and configure them in the same way you would for any other Unifi message. Note: the field maps will likely need to be specific to the dataset fields.

<figure><img src="https://2763208740-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZR0dkdEH0Mm7amiIzsdG%2Fuploads%2FT9jjb6QosdZjiKhtgW3H%2Fimage.png?alt=media&#x26;token=0c5119b2-b281-411d-ade9-9ba95df8b555" alt=""><figcaption><p>Fields for Dataset mapping in the process message</p></figcaption></figure>

Use the Coalesce field (specific to Dataset messages) to indicate which field should be used for identifying existing records to update during inbound processing.