Flat File Source Connector

The Flat File source connector in DataSync lets you retrieve data from structured text files such as CSV and TXT for loading or synchronizing in your data warehouse. For large files, you can adjust row scanning or override automatic detection using a custom Schema.ini file. You can also connect to local, network, or cloud‑hosted CSV resources by setting the URL parameter and any required authentication properties.

After creating all required source connections, configure your destination source to complete the connection setup.

Create a source connection in DataSync

1. Log in to DataSync.
2. From the welcome screen, select Connections.
3. Next to Source Connections, click New.
4. Select Flat File.
5. In the Connection Properties panel, enter the connection properties.
6. (Optional) In the Additional Connection Properties panel, select Add property and enter the parameters for each property.
7. In the Advanced Settings panel, configure the settings, including the Tracking Type and other values according to your requirements.
8. Click Save.

Parameters

Connection properties

Parameter	Description
Description	Unique name for the connection. Example: Flat File
URL	Location of source file or folder. Supports local or network paths and can point to a single CSV file or a folder with multiple CSVs. Examples: C:\Data\SalesData.csv or \\Server\Share\Data
Include Types	File types to include: CSV, TXT, or both.
Delimiter	Format used for parsing: Comma, Tab, or Custom.
Custom Delimiter	Character used when delimiter is set to Custom. Example: ;
Quote Character	Character used to quote values in the file.
Top Rows to Skip	Number of rows to skip at the top of the file.
Exclude file extension	Removal of the file extension from table names during extraction.
First row contains header	Header row setting where the first row is treated as column headers.
Include subdirectories	Inclusion of files and schema.ini from nested folders.
Use row numbers	Addition of a RowNumber column as a key when updating or deleting CSV files.
Verbosity	1 – Log queries, row counts, execution start/end, errors. 2 – Includes level 1 plus cache queries, HTTP headers. 3 – Includes level 2 plus request/response bodies. 4 – Includes level 3 plus transport-level communication. 5 – Includes level 4 plus all interface commands.
Enable Pooling	Connection pooling option for performance.
Pool idle timeout	Maximum idle time for connections before returning them to the pool, in seconds.
Max Pool Size	Maximum number of connections allowed in the pool.
Pool wait time	Maximum wait time for connection allocation before error is thrown, in seconds.

Additional connection properties

Additional connection string properties not specified in the Connection Properties panel. For each property added, you can choose Visible or Encrypted. Selecting Encrypted hides the value from the interface and stores it encrypted in the back end, such as when defining passwords. Typical properties are listed below.

Property	Value
Charset	Character encoding for the source file when encoded in ANSI. No parameter is needed for UTF‑8 encoded files. Example: windows-1252
Culture	Regional format determining decimal and thousands separators. Must match between source/destination files and DataSync configuration. Examples: fr-FR for commas or en-US for periods.
RowScanDepth	Number of rows scanned to determine table column data types. Default: 100. Examples: 5000 to scan more rows for accuracy. 0 scans the entire file.
AggregateFiles	All files combined from the URI directory into a single table called AggregatedFiles. Default schema is from the first file unless MetadataDiscoveryURI is set. Example: aggregating multiple product inventory CSVs.
IncludeFiles	Comma-separated list of allowed file extensions. Default: CSV,TXT,TAB. Supports NOEXT for files without extensions and archives (ZIP, TAR, GZ). Can also filter by file masks and date/time attributes. Examples: IncludeFiles="TXT,CreatedDate<='2020-11-26T07:39:34-05:00'" or IncludeFiles=2020*.csv,TXT
ExcludeFiles	Comma-separated list of excluded file extensions. Supports date/time filters similar to IncludeFiles. Example: ExcludeFiles="TXT,CreatedDate<='2020-11-26T07:39:34-05:00'"
InsertMode	Mode for inserting into CSV files: FilePerBatch – Creates a new CSV per batch (mandatory for NEC Cloud). SingleFile – Inserts all data into one file, for local/network paths only.
CreateBatchFolder	Option to create a folder when InsertMode is set to FilePerBatch. Default: true.
BatchNamingConvention	Naming format for batch files: Timestamp_BatchNumber – yyyyMMddhhmmssSSS_batchNumber. TableName_BatchNumber – TableName_batchNumber.csv (default).

Advanced settings

Advanced settings control how the Flat File connector tracks changes, handles regional and time configuration, and processes data batches during extraction. These options allow fine‑tuning for performance and accuracy, and should be configured according to your system environment and operational requirements.

Setting	Description
Tracking Type	Method for tracking changes: None or Date.
Region	Region setting for the connector, if required by your setup.
Time Zone	Time zone matching the Flat File application server.
Time Offset	Refresh offset in seconds to compensate for timing issues in record selection. Minimum value is 0; maximum is 3600 seconds.
Batch Size	Quantity of records processed in each batch during extraction. Larger batch sizes increase memory usage but can improve performance up to a point. The default value is 2000 and the maximum should not exceed 10000 records. Adjust according to your network speed and disk performance; in most cases the default (2000) works best.

Example