With the introduction of Datasets, it is now possible for large amounts of user, hardware, network and other supporting data to be sent to and received from remote systems. While this has been possible for some time using Pollers, they are not specific to handling large amounts of data in the way that Datasets are.

Datasets provide easy configuration that is supported by platform intelligence and automation, making it is incredibly easy to setup a robust and efficient mechanism for handling large sets of data.

Integration Build

Each Dataset will create the following configuration:

• The Send message is used for sending the data in an attachment to the other system. The name is automatically generating using the prefix "Send" and the name of the table.

• The Process message is used for processing the record data both inbound and outbound. The name is automatically generating using the prefix "Process" and the name of the table.

• The Import set table is used for staging inbound data. The name is automatically generated using the prefix "dataset" with the sys_id of the dataset.

• The Scheduled import record is automatically created for processing import data. It is executed when an inbound Dataset Request has created the import set rows and is ready to be processed.

• The Transform map and associated coalesce fields are used for processing the import set.

Outbound Data

Data is automatically collected, transformed, packaged and sent to another system on a schedule. Each export creates one or more Dataset Requests depending on the number of records and a series of limits configurable on the Dataset.

Unifi automatically creates a Process message which can be used with outbound Fields and Field maps (via the Source to Stage and Stage to Request Message Scripts) to extract the data from the specified records.

The extracted data is written to an attachment which is then sent using the Send message which is also automatically created by Unifi.

Inbound Data

When an inbound Send message (as specificed on the Dataset) with a data attachment is received, the attachment will be processed with each record being inserted into an import set table directly related to the Dataset. The import set table is automatically created and maintained during the Integration Build process.

Unifi uses ServiceNow Import Sets since they offer a well understood mechanism for importing data with performance benefits. The mapping is automatically managed by Unifi and handled through a transform script which uses inbound Fields and Field maps (via the Stage to Target Message Script) to transform the data and write it to the target records.

Fields

The process message related to the Dataset is used for inbound and outbound mapping and transformation.

Simply create the fields that you want to export/import and configure them in the same way you would for any other Unifi message. Note: the field maps will likely need to be specific to the dataset fields.

Use the Coalesce field (specific to Dataset messages) to indicate which field should be used for identifying existing records to update during inbound processing.

Creating a new Dataset for exporting and importing data is a simple 3-step process.

1. Create a new Dataset

2. Configure the Process Message

3. Configure the Send Message

4. Setup an export schedule

5. Build

Create a New Dataset

1. Open Integration Designer and select or create an Integration.

2. From the main integration menu, select Datasets.

3. Click New.

4. Fill in the required information and choose your configuration.

5. Click Submit and view.

Creating a new Dataset will automatically configure several dependencies in your instance. These are:

• A Message called Process_<table>

• A Message called Send_<table>

• A Scheduled Import Set with the same name as the Dataset

• A Transform Map

Configure Process Message

The Process Message handles the data mapping. Use the Fields list to configure which fields are being imported/exported.

1. From the Dataset details tab, navigate to the Process message and use the clickthrough button to open it.

2. From the Message, open the Fields list and add the fields that should be imported/exported. Each field will need to have a Dataset specific Field Map for transforming the data.

3. Ensure at least one field will Coalesce so the import can match to existing records.

4. Click Build Message.

Configure Send Message

The Send Message handles the data to be imported or exported as an attachment.

The Path will need to be configured depending on where the data is being sent. If you are connecting to another ServiceNow instance using a ShareLogic endpoint, you will likely need to configure the Path to use the dataset web service at path "/dataset".

1. From the Dataset details tab, navigate to the Send message and use the clickthrough button to open it.

2. Open the Outbound > Settings page.

3. Configure the Path as required, e.g. "/dataset".

4. Click Save.

Setup an Export Schedule

Datasets have the same schedule logic as any other scheduled job and can be configured to export data whenever needed.

1. Open the Dataset.

2. Click Scheduling to open the scheduling tab.

3. Configure the desired schedule.

4. Click Save.

Build

Several important features are built into the integration build process for Datasets. Ensure you build the integration whenever you have finished adding or making major changes to a Dataset.

1. Scheduled Import Sets are created and updated.

2. Transform Maps are created and updated.

Import Logging

Inbound Datasets use ServiceNow Import Sets giving you the ability to view dataset imports as you would any other data import.

For detailed Activity Logs, navigate to the Transform page on a Dataset and enable the Import logging option.

Dataset Cleanup

Dataset Requests are a record of when all Dataset imports and exports occurred. They are linked directly to Transactions that belong to a Bond which is exclusive to the Dataset. The records are automatically deleted when the corresponding Transaction is deleted.

The number of Transactions to keep during the daily cleanup can be managed using the Cleanup option which can be configured on each Dataset. This value specifies the number of transactions that should remain following the cleanup process. The default retention is 10 records.

Any Dataset Requests which are somehow orphaned, meaning they have no Dataset or no Transaction specified, will be removed based on the Orphan Record Cleanup Age x_snd_eb.orphan_record.cleanup.age system property. The default orphan cleanup age is 7 days.

Export Tuning

When a Dataset export runs, the system will automatically export the records in small batches depending on its configuration.

Batch size options are:

• Max size: set the maximum file size to export. There is an internal maximum size of 5MB.

• Max rows: limit batches to a specific number of records.

• Max time: prevent long running exports by setting a maximum export time.

The final export size is determined by whichever limit is reached first. If more records are available to be processed, additional Dataset Requests will be created until all records have been exported.

Create a Dataset Field Map

New Field Maps can be configured for handling specific types of data in new ways. We recommend using the "Dataset" prefix when creating your new field map so it is easy to identify.

Dataset specific Field Maps are slightly different to eBonding integration field maps. The main difference is that the Stage to Target script is executed as a ServiceNow Transform Map script. This means that certain objects are not available, including transaction, bond, and request.

These Field Maps should reference the Import Set Row fields using the standard $stage object. Field name conversion from the Import Set Row source object to the Field Map $stage object is handled automatically, i.e. "u_" prefixes are removed.

The target object which represents the record being created or updated can be used normally.

The log object is updated to reference ws_console meaning all logs will be written to the Activity Log generated for each imported record. If required, logs can continue to be written to the Import Set Row using the source object.

Disable Dataset Import

By default, Datasets are automatically configured for both import and export. If you require a one-way data export without import, you can prevent inbound messages from being processed.

1. From Integration Designer, navigate to the Dataset in your integration.

2. Find the Send message and click the clickthrough button to open it.

3. Change the Direction from Bidirectional to Outbound.

4. Click Save.

Multiple Tables

Datasets are designed to work with one table at a time. If you need to import/export more than one table, setup a Dataset for each table.

Sending Data from a Third Party

Externally generated files can be processed by a Dataset. Files should match the file type the Dataset expects, e.g. CSV or JSON, and be streamed to the Dataset import endpoint. The maximum file size will depend on your instance configuration.

POST https://<instance>.service-now.com/api/x_snd_eb/unifi/<api_name>

Send a file to be processed by a Dataset.

Example: https://acme.service-now.com/api/x_snd_eb/unifi/incident/dataset?file_name=cmdb_ci.csv&reference=Sync%20Server

Query Parameters

Headers

Using IRE (Identification and Reconciliation Engine)

It is possible to add support for ServiceNow IRE to your Dataset imports. This will push data through the reconciliation engine rather than directly updating the target records from the Import Set.

Follow these steps to configure your Dataset for use with IRE.

1. Create a new Data Source.

2. Create a Field Map for setting up the import for IRE.

3. Create a Field Map to give the data to IRE.

4. Add header and footer fields to the Dataset.

5. Modify Field Maps being used by the Dataset.

var dsUtil = new global.CMDBDataSourceUtil();
dsUtil.addDataSource("ACME CMDB");

// Setup an IRE item for this record
$stage.$ire_item = {
  className: target.getTableName(),
  internal_id: '' + target.sys_id,
  values: {},
  lookup: [],
  sys_object_source_info: {
    source_feed: '$[field.integration.name] Dataset ' + target.getTableName(),
    source_name: '$[field.integration.name]',
    source_native_key: '' + target.sys_id,
    source_recency_timestamp: '' + new GlideDateTime()
  }
};

var input = JSON.stringify({ items: [$stage.$ire_item] }, null, 2);

x_snd_eb.ws_console.debug('Payload for IRE is:\n\n %0', [input]);

var source_name = $stage.$ire_item.sys_object_source_info.source_name;
var output = sn_cmdb.IdentificationEngine.createOrUpdateCI(source_name, input);

x_snd_eb.ws_console.debug('Output from IRE:\n\n {0}', [output]);

ignore = true;

// Store field values for IRE
if ($stage.$ire_item) {
  $stage.$ire_item.values['$[field.element]'] = target.getDisplayValue('$[field.element]');
}

// Store field values for IRE
if ($stage.$ire_item) {
  $stage.$ire_item.values['$[field.element]'] = target.getDisplayValue('$[field.element]');

  // Add the lookup object for reference fields
  $stage.$ire_item.lookup.push({
    className: target.$[field.element].getReferenceTable(), 
    values: {
      name: '' + target.$[field.element].getDisplayValue()
    }
  });
}

Publishing with Run As user

When publishing Datasets (or any scheduled script) to an update set or application, especially if submitting to the ServiceNow Store, you need to ensure the "Run as" field is emptied since the user will likely not exist on the target system.