# Dataset Extras

## Import Logging

Inbound **Datasets** use **ServiceNow Import Sets** giving you the ability to view dataset imports as you would any other data import.

For detailed **Activity Logs**, navigate to the **Transform** page on a **Dataset** and enable the **Import logging** option.

{% hint style="warning" %}
Datasets with **Import logging** enabled will generate one **Activity Log** for each **Import Set Row**.
{% endhint %}

## Dataset Cleanup

**Dataset Requests** are a record of when all **Dataset** imports and exports occurred. They are linked directly to **Transactions** that belong to a **Bond** which is exclusive to the **Dataset**. The records are automatically deleted when the corresponding **Transaction** is deleted.

The number of **Transactions** to keep during the daily cleanup can be managed using the **Cleanup** option which can be configured on each Dataset. This value specifies the number of transactions that should remain following the cleanup process. The default retention is 10 records.

Any **Dataset Requests** which are somehow orphaned, meaning they have no **Dataset** or no **Transaction** specified, will be removed based on the **Orphan Record Cleanup Age** `x_snd_eb.orphan_record.cleanup.age` system property. The default orphan cleanup age is 7 days.

## Export Tuning

When a **Dataset** export runs, the system will automatically export the records in small batches depending on its configuration.

Batch size options are:

* **Max size:** set the maximum file size to export. There is an internal maximum size of 5MB.
* **Max rows:** limit batches to a specific number of records.
* **Max time:** prevent long running exports by setting a maximum export time.

The final export size is determined by whichever limit is reached first. If more records are available to be processed, additional **Dataset Requests** will be created until all records have been exported.

## Create a Dataset Field Map

New **Field Maps** can be configured for handling specific types of data in new ways. We recommend using the "Dataset" prefix when creating your new field map so it is easy to identify.

**Dataset** specific **Field Maps** are slightly different to eBonding integration field maps. The main difference is that the **Stage to Target script** is executed as a **ServiceNow Transform Map** script. This means that certain objects are not available, including `transaction`, `bond`, and `request`.

These **Field Maps** should reference the **Import Set Row** fields using the standard `$stage` object. Field name conversion from the Import Set Row `source` object to the **Field Map** `$stage` object is handled automatically, i.e. "u\_" prefixes are removed.

The `target` object which represents the record being created or updated can be used normally.

The `log` object is updated to reference `ws_console` meaning all logs will be written to the **Activity Log** generated for each imported record. If required, logs can continue to be written to the **Import Set Row** using the `source` object.

## Disable Dataset Import

By default, **Datasets** are automatically configured for both import and export. If you require a one-way data export without import, you can prevent inbound messages from being processed.

1. From **Integration Designer**, navigate to the **Dataset** in your integration.
2. Find the **Send message** and click the clickthrough button to open it.
3. Change the **Direction** from **Bidirectional** to **Outbound**.
4. Click **Save**.

## Multiple Tables

**Datasets** are designed to work with one table at a time. If you need to import/export more than one table, setup a **Dataset** for each table.

## Sending Data from a Third Party

Externally generated files can be processed by a **Dataset**. Files should match the file type the **Dataset** expects, e.g. **CSV** or **JSON**, and be streamed to the **Dataset** import endpoint. The maximum file size will depend on your instance configuration.

<mark style="color:green;">`POST`</mark> `https://<instance>.service-now.com/api/x_snd_eb/unifi/<api_name>`

Send a file to be processed by a Dataset.

Example: <https://acme.service-now.com/api/x\\_snd\\_eb/unifi/incident/dataset?file\\_name=cmdb\\_ci.csv\\&reference=Sync%20Server>

#### Query Parameters

| Name                                         | Type   | Description             |
| -------------------------------------------- | ------ | ----------------------- |
| file\_name<mark style="color:red;">\*</mark> | String | Name of the file        |
| reference<mark style="color:red;">\*</mark>  | String | The name of the dataset |

#### Headers

| Name                                                    | Type   | Description                                       |
| ------------------------------------------------------- | ------ | ------------------------------------------------- |
| Content-Type<mark style="color:red;">\*</mark>          | String | text/csv OR application/json                      |
| x-snd-eb-message-name<mark style="color:red;">\*</mark> | String | The name of the Send message, e.g. Send\_\<table> |

{% tabs %}
{% tab title="200: OK " %}

{% endtab %}
{% endtabs %}

## Using IRE (Identification and Reconciliation Engine)

It is possible to add support for **ServiceNow IRE** to your **Dataset** imports. This will push data through the reconciliation engine rather than directly updating the target records from the **Import Set**.

Follow these steps to configure your **Dataset** for use with IRE.

1. Create a new **Data Source**.
2. Create a **Field Map** for setting up the import for IRE.
3. Create a **Field Map** to give the data to IRE.
4. Add header and footer fields to the **Dataset**.
5. Modify **Field Maps** being used by the **Dataset**.

<details>

<summary>Create a new Data Source</summary>

A new **Data Source** is required for IRE rules to be configured. You can add this manually to the sys\_choice table on the discovery\_source field, or create it using this fix script.

We recommend the data source name matches the integration name. This is how the IRE header **Field Map** is setup in this example.&#x20;

If you are packaging Datasets in Scoped Applications, you should create a **Fix Script** that runs when the application is installed.

```javascript
var dsUtil = new global.CMDBDataSourceUtil();
dsUtil.addDataSource("ACME CMDB");
```

</details>

<details>

<summary>Create a Field Map for setting up the import for IRE.</summary>

#### Name

Use IRE - Header

#### Description

Sets up a Dataset Import for using the ServiceNow IRE (Identification and Reconciliation Engine) to import CMDB data.&#x20;

Do not map fields using this field map to a CMDB attribute. Ensure that it is the first field to be processed by setting the field order to an appropriately small number.

This field map will create an $ire\_values object on the $stage which will store all the values to give to IRE.

#### Stage to Target

```javascript
// Setup an IRE item for this record
$stage.$ire_item = {
  className: target.getTableName(),
  internal_id: '' + target.sys_id,
  values: {},
  lookup: [],
  sys_object_source_info: {
    source_feed: '$[field.integration.name] Dataset ' + target.getTableName(),
    source_name: '$[field.integration.name]',
    source_native_key: '' + target.sys_id,
    source_recency_timestamp: '' + new GlideDateTime()
  }
};
```

</details>

<details>

<summary>Create a <strong>Field Map</strong> to give the data to IRE.</summary>

#### **Name**

Use IRE - Footer&#x20;

#### **Description**

Use the ServiceNow IRE (Identification and Reconcilliation Engine) to import CMDB data.

Do not map fields using this field map to a CMDB attribute. Ensure that it is the last field to be processed by setting the field order to an appropriately large number.

This field map will convert the inbound payload to the IRE format, call the createOrUpdateCI() function, and ignore the current import set row.

#### **Stage to Target**

```javascript
var input = JSON.stringify({ items: [$stage.$ire_item] }, null, 2);

x_snd_eb.ws_console.debug('Payload for IRE is:\n\n %0', [input]);

var source_name = $stage.$ire_item.sys_object_source_info.source_name;
var output = sn_cmdb.IdentificationEngine.createOrUpdateCI(source_name, input);

x_snd_eb.ws_console.debug('Output from IRE:\n\n {0}', [output]);

ignore = true;
```

</details>

<details>

<summary>Add header and footer Fields to the Dataset.</summary>

1. Navigate to the **Process message** from the Dataset.
2. Add a new **Field**
   * **Map to field** is **false**
   * **Field map** is **Use IRE – Header**
   * Call the property **IRE\_Header**
   * Set the **Order** to be **1** or less (it needs to be the first field on the message)
3. Add a new **Field**
   * **Map to field** is **false**
   * **Field map** is **Use IRE – Footer**
   * Call the property **IRE\_Footer**
   * Set the **Order** to be **10000** or more (it needs to be the last field on the message)

</details>

<details>

<summary>Modify Field Maps</summary>

Each **Field** needs to registered for processing by IRE. This is achieved by adding the following snippets (depending on field type) to the end of the **Stage to Target** script for each **Field Map** being used in the **Dataset**.

#### Standard Field Map - Stage to Target

Most fields only need to tell IRE the field name and the value.

<pre class="language-javascript"><code class="lang-javascript">// Store field values for IRE
if ($stage.$ire_item) {
<strong>  $stage.$ire_item.values['$[field.element]'] = target.getDisplayValue('$[field.element]');
</strong><strong>}
</strong></code></pre>

#### Reference Field Map - Stage to Target

Reference fields need an additional lookup object to pass into IRE.

<pre class="language-javascript"><code class="lang-javascript">// Store field values for IRE
if ($stage.$ire_item) {
<strong>  $stage.$ire_item.values['$[field.element]'] = target.getDisplayValue('$[field.element]');
</strong><strong>
</strong>  // Add the lookup object for reference fields
  $stage.$ire_item.lookup.push({
    className: target.$[field.element].getReferenceTable(), 
    values: {
      name: '' + target.$[field.element].getDisplayValue()
    }
  });
}
</code></pre>

</details>

{% hint style="info" %}
Remember to **Build** the Message/Integration when you have finished configuring the **Fields**.
{% endhint %}