Is it possible to configure dependent fields connected to a CSV datasource?


Table of Contents



It depends

File and URL datasource: No

When using a File or a URL datasource connected to a CSV file, it's not possible.

Elements Connect does not provide any way to query children CSV files content.

JNDI datasource: Yes

With a JNDI datasource, you can use an open source CSV JDBC driver to query your CSV files: csvjdbc.

The benefits brought by this solution are:



use the power of SQL on your datasource file (WHERE, GROUP BY, ORDER BY, ...)


See the structure of your CSV files using the datasource explorer - every file is a distinct table




Performances

While this datasource type requires few configuration steps (no need to configure a database), it shall not be used on very large datasets.

We've made some tests on CSV files with 14 columns, with a Jira 8.2 instance running on a 2016 Macbook Pro with a SSD drive.

The query used for our tests was:

SELECT *
FROM <FILE>


# linesAverage query time
10

2 MS

100

3 MS

1,000

9 MS

5,000

50 MS

10,000

115 MS

50,000

390 MS

100,000

741 MS

500,000

4000 MS

1,000,000

8800 MS

1,500,000

13000 MS

That's a constant average of 9 μs (10-6 sec) per row (except for small files where file opening / closing seems to generate a significant overload)


Configuration

Have a look at the JNDI Datasource page to learn how to configure a CSV SQL datasource.