Context We’re looking to build a Teradata source connector. A member of our Contributor Program @Ivica Taseski will be driving this forward and folks from Teradata @Satish Chinthanippu @Adam Tworkiewicz have agreed to support any questions that arise.

Everyone else in this channel is from Airbyte as an FYI throughout the process and who might be able to jump in if Airbyte needs to weigh in.

Airbyte has joined this channel by invitation from Airbyte Team.

Hi @Ivica Taseski, it’ s nice to meet you. I’m a product manager here at Teradata for a few different products, including our integration with Airbyte. I’ve heard that you are interested in developing a Teradata Airbyte source. Let us know what use case you have in mind and if you already have any idea what volumes the source would have to support.

@Adam Tworkiewicz to clarify, Ivica is helping us develop it broadly for requests we’ve seen come in. The use case is getting sales and finance tables from Teradata Vantage and looking at about 20GB of data volumes daily.

ah, thanks 🙂

Greetings everyone! For starters we will need access to Teradata for integration testing purposes. What is the go to approach here? Teradata account, docker image or something else?

It will be an account. I’ve shared credentials with Greg Solovyev. If you prefer a separate test account, let me know.

Small update. I have managed to create an environment/database using the provided API and I have connected to it using the JDBC driver. Onto the development of the connector next.

Hey @Greg Solovyev (Airbyte) @Satish Chinthanippu octavia wave The Http client for managing Teradata environments can be found here (https://github.com/itaseskii/airbyte/tree/teradata-source/airbyte-integrations/connectors/source-teradata/src/test-integration/java/io/airbyte/integrations/source/teradata/client). The entrypoint for the client is the TeradataHttpClient class which implements the following methods with examples on how to use them:

POST {base_url}/environments

TeradataHttpClient teradataHttpClient = new TeradataHttpClient("<https://api.clearscape.teradata.com>");
var request = new CreateEnvironmentRequest("name", Region.US_CENTRAL.getRegionName(), "password");
var response = teradataHttpClient.createEnvironment(request, "api-token");
// make sure to call get() otherwise the background/daemon thread gets shut down before the http call is made
response.get();

GET {base_url}/environments/{environment_name}

TeradataHttpClient teradataHttpClient = new TeradataHttpClient("<https://api.clearscape.teradata.com>");
var request = new GetEnvironmentRequest("name");
var response = teradataHttpClient.getEnvironment(request, "api-token");

DELETE {base_url}/environments/{environment_name}

TeradataHttpClient teradataHttpClient = new TeradataHttpClient("<https://api.clearscape.teradata.com>");
var request = new DeleteEnvironmentRequest("name");
var response = teradataHttpClient.deleteEnvironment(request, "api-token");
// make sure to call get() otherwise the background/daemon thread gets shut down before the http call is made
response.get();

With that in mind it should be pretty straightforward to make the necessary calls in the

@BeforeAll

and

@AfterAll

test hooks. P.S There is another method named

pollingCreateEnvironment

which is currently not implemented in order to unblock you faster but the main purpose of that method is to avoid long running connections/operations on

POST {base_url}/environments

and

DELETE {base_url}/environments/{environment_name}

which take ~1.5 min on average and can cause connection issues and instead check for instance readiness by polling the

GET {base_url}/environments/{environment_name}

endpoint repeatedly until the instance goes in status RUNNING.

octavia wave Okay, so I have come across an issues which I can't overcome. I have tried using both the terajdbc4 (teradata vantage driver) and terajdbc (should I be even using it?) drivers with latest versions and I keep getting

Receiver class com.teradata.jdbc.jdk6.JDK6_SQL_ResultSet does not define or inherit an implementation of the resolved method 'abstract java.lang.Object getObject(int, java.lang.Class)' of interface java.sql.ResultSet.

java.lang.AbstractMethodError: Receiver class com.teradata.jdbc.jdk6.JDK6_SQL_ResultSet does not define or inherit an implementation of the resolved method 'abstract java.lang.Object getObject(int, java.lang.Class)' of interface java.sql.ResultSet.

at com.zaxxer.hikari.pool.HikariProxyResultSet.getObject(HikariProxyResultSet.java)

which is most likely caused by the JDBC driver being incompatible with newer versions of Java (17 being used in the connector) since the ResultSet interface includes a method getObject(int, java.lang.Class) which isn't implemented by the driver. Is there some workaround for this other than reimplementing the source abstractons to use a supported getObject() method and type casting the ResultSet myself? Imo downgrading to a lower version of Java in the connector shouldn't be even considered both from maintenance and performance reasons.

@Satish Chinthanippu @Adam Tworkiewicz can you help here ☝🏼 ?

btw I have just tried terajdbc 20.00.00.02 and 20.00.00.06 and going through the internals the implementation for

java.lang.Object getObject(int, java.lang.Class)

seems to be present but implemented by always returning null.

@Ivica Taseski I just checked with a sample with teradata JDBC drivers - 17.20.00.12 and 20.00.00.06 and with Java 17 and able to get value through getObject for JSON db column type

image.png

I am not sure for which column types and scenario you are trying. If you provide more details, I can check and confirm with my jdbc team if any help required

Quick question related to result ordering in Teradata. Let's say I perform the following inserts in the db

INSERT INTO table_name (name, tmstmp) VALUES ('c', '2021-01-02 00:00:00')

INSERT INTO table_name (name, tmstmp) VALUES ('d', '2021-01-02 00:00:00')

INSERT INTO table_name (name, tmstmp) VALUES ('e', '2021-01-02 00:00:00')

INSERT INTO table_name (name, tmstmp) VALUES ('f', '2021-01-03 00:00:00')

and I execute a query for retrieving the above data with three identical timestamps and one not

SELECT * FROM table_name WHERE tmstmp > 2021-01-01 00:00:00 ORDER BY tmstmp ASC

shouldn't entries be ordered by insertion order when equal ordering values or does Teradata make no such guarantees? What I'm getting is [d, c, e, f] instead of [c, d, e, f]

✅

Just FYI here, Andy Yeo and I are no longer at AirByte.

@Ivica Taseski i will be your point of contact for this. Let me know if you need more info on getting the integration tests working. (related to my comment on your PR)

@Ivica Taseski how much time did you estimate to create the Teradata source?