This article is the guide on how to use Inbound Data Migration (Data ingestion) for data files in Trellispark. Data Migration allows developers to import data to the platform by simply inserting the desired file and extracting the required information seamlessly. This simplifies the process since migrated data integration (shaping and preparing data to be consumable in our platform) is already taken care of.
Step 1: Preparing files for import
Before you begin migrating your file for import, ensure all the data you require is within their proper rows and columns. In the example below, we have a total of 4 rows and 8 columns with varying data types. Keep in mind the location of data and their types as that will be important in step 2.
Step 2: Preparing Integration Targets
Once your file is ready to go, you’ll now want to specify your spreadsheet columns for your integration targets. To get there, you will go to your Upload file functionality and choose the appropriate Form concept you will be using. In this example, we will use our Dealership Concept in our Forms tab:
From there you will want to navigate to you Data model Tab and choose the appropriate Definition, in this example it is the field definition labelled UploadFile. Once chosen, scroll down to the bottom where you will see Integration Targets. Here is where you can add new integration targets and update ones that currently exist :
For this example, lets choose our Car Workspace Integration Target. Lets take a look at what options we have once inside:
There are multiple important options you can choose from within this screen:
- Name of the integration target. This can be whatever you’d like.
- File type of Inbound data file
- Direction of file (Inbound/Outbound)
- Concept used for integration target. Below is the example information from the Car concept used for the columns for clarity:
- Allowing overwrites of data(yes/no)
- Parent concept if applicable. In this example we don’t have one so it remains empty.
- Primary key and Primary key of parent record. In our example our Primary key belongs in column 7 so we’ve chosen 7 for this section.
- Whether the first row has labels for your columns and if it does, you have the option to ignore x rows if necessary.
- Delimiter used for column separation within your file. For our example, we’ll use a ‘,’ separator.
- Integration target types (API, TSQL, Datamodel)
- Default Parent (Instance, Parent, Specified, Workspace)
At the bottom of the screen, you’ll find the Spreadsheet Columns section. This is where you specify details for each column: their XMLElementPath names derived from the previously selected concept (in our case, TestUploadFile.Car), the data types for migration, and whether null values are allowed. For instance, in our CSV, the “Date” column comes first, so we select column 1 and assign it the DateTime data type. By default, if the “Allow Nulls” option is left blank, it is set to “Yes.”
Once you’ve defined all the appropriate information based on your file, you can click save and continue creating all columns required with the appropriate information.
Step 3: Importing file for Inbound Data Migration
Once you’ve completed step 1 & 2, its time to start importing! Within your Application form you’ll find the field definitions you’ve defined and customized. In this particular example, we have created a Dealership and Cars childlist that will hold the data from the file after it has been migrated. Lets create a new Dealership:
Here we’re given the ability to name the dealership, upload the appropriate file for import and manually add new cars if necessary.
Once the file has been added, you’re given the choice of integration targets. Choose whichever is appropriate and click load. For our example, we will choose the Car Instance.
You’ll then be prompted to ensure you’ve chosen the right integration target:
Once accepted it will update you whether or not the file was successful. Here we can see the data was successful migrated and the appropriate information from the CSV file was added to the ChildList of cars on the right:
Alternatively, If your data doesn’t meet the specified column parameters, you’ll get a notification and a “View Errors” button that you can click and review the issue. Here we’ve intentionally changed an Integer to a String as seen within the error context. Despite errors, any other data that matches the column criteria will continue to be processed, ensuring that a single error doesn’t stop the entire migration. You can then address any discrepancies manually by following the information provided in the “View Errors” pop-up window.
We can also take a look at each item and see that the information was added in the appropriate area. All of this information can also be manually changed at any time if necessary.
Note: If you’ve chosen to allow overwrites and the data contains a Primary key, the data can only be stored within one unique integration target. To allow for duplicates in multiple different targets, ensure overwrites are turned off and unlink any primary key associated with the desired data. Otherwise all unique data can be stored in any integration target.
