Flat File Source Connector

The Flat File source connector lets DataSync retrieve data from structured text files such as CSV and TXT and load it into your data warehouse. For large files, you can adjust row scanning or override automatic detection using a custom Schema.ini file. You can also connect to local, network, or cloud‑hosted CSV resources by setting the URL parameter and any required authentication properties.

Once your source connection is ready, configure your destination connection to finish the setup.

Create the source connection in DataSync

1. Log in to DataSync.
2. From the welcome screen, select Connections.
3. Next to Source Connections, click New.
4. Select Flat File.
5. Enter all required connection properties.
6. (Optional) In the Additional Connection Properties panel, click Add property and enter any extra parameters you need.
7. Configure the advanced settings to match your environment, including Tracking Type.
8. Click Save.

Connection properties

Property	What to enter
Description	Unique name for the connection. Example: Flat File.
URL	Path to the source file or folder. Supports local paths, network shares, and can point to a single CSV file or a folder containing multiple files. Example: C:\Data\SalesData.csv or \\Server\Share\Data.
Include Types	File types to include: CSV, TXT, or both.
Delimiter	Character used to separate values: Comma, Tab, or Custom.
Custom Delimiter	Character used when Delimiter is set to Custom. Example: ;.
Quote Character	Character used to wrap quoted values in the file.
Top Rows to Skip	Number of rows to skip at the top of the file before reading data.
Exclude file extension	Removes the file extension from table names during extraction.
First row contains header	Treats the first row as column headers.
Include subdirectories	Includes files and schema.ini from nested subfolders.
Use row numbers	Adds a RowNumber column as a key when updating or deleting CSV records.
Verbosity	Controls how much detail the connector writes to the log. Each level includes everything from the level below it, plus additional detail. 1 logs queries, row counts, execution start and end times, and errors. 2 adds cache queries and HTTP headers. 3 adds request and response bodies. 4 adds transport-level communication. 5 adds all interface commands.
Enable Pooling	Enables connection pooling, which keeps a set of database connections open and reuses them across extractions instead of opening a new connection each time. This reduces overhead and improves performance when multiple extractions run at the same time.
Pool idle timeout	Time in seconds a connection can stay idle before returning to the pool.
Max Pool Size	Maximum number of connections allowed in the pool at the same time.
Pool wait time	Time in seconds DataSync waits for an available connection before throwing an error.

Additional connection properties

Use this panel to enter connection string properties not available in the Connection Properties panel. For sensitive values such as passwords, set the type to Encrypted. The value is hidden in the interface and stored encrypted in the back end.

Property	What to enter
Charset	Character encoding for the source file when it uses ANSI encoding. Not needed for UTF-8 files. Example: windows-1252.
Culture	Regional format that determines how decimal and thousands separators are interpreted. Must match between source files and your DataSync configuration. Example: fr-FR for comma decimals or en-US for period decimals.
RowScanDepth	Number of rows scanned to infer column data types. Default: 100. Use 5000 for better accuracy on varied data, or 0 to scan the entire file.
AggregateFiles	Combines all files from the source folder into a single table called AggregatedFiles. The schema is taken from the first file unless MetadataDiscoveryURI is set. Useful when working with multiple files that share the same structure, such as monthly export files.
IncludeFiles	Comma-separated list of file extensions to include. Default: CSV,TXT,TAB. Supports files without extensions (NOEXT), archives (ZIP, TAR, GZ), file masks, and date filters. Example: IncludeFiles=2020*.csv,TXT.
ExcludeFiles	Comma-separated list of file extensions to exclude. Supports the same date and mask filters as IncludeFiles. Example: ExcludeFiles=TXT.
InsertMode	Controls how data is written to CSV files. FilePerBatch creates a new CSV file per batch. Required for NEC Cloud. SingleFile writes all data into one file. For local and network paths only.
CreateBatchFolder	Creates a folder when InsertMode is set to FilePerBatch. Default: true.
BatchNamingConvention	Naming format for batch files. Timestamp_BatchNumber uses the format yyyyMMddhhmmssSSS_batchNumber. TableName_BatchNumber uses the format TableName_batchNumber.csv. This is the default.

Advanced settings

These settings control how the connector tracks data changes, handles time and regional configuration, and processes records during extraction. Configure them to match your Flat File environment so that results stay accurate and consistent.

Setting	What to select
Tracking Type	Method for tracking data changes: None or Date.
Region	Region setting for the connector, if required by your setup.
Time Zone	Time zone matching your Flat File environment.
Time Offset	Refresh offset in seconds to compensate for timing issues in record selection. Minimum 0, maximum 3600.
Batch Size	Number of records processed per batch during extraction. Larger batches can improve performance but use more memory. Default is 2000, maximum is 10000. Adjust based on your network speed and disk performance. The default works well in most cases.

Example setup

Completed Flat File source connection in DataSync, with all properties and settings filled in.