anyone, I'm trying to follow Robin's blog how to c...
# troubleshooting
g
anyone, I'm trying to follow Robin's blog how to create some persistance with flink...
Copy code
https://www.decodable.co/blog/catalogs-in-flink-sql-hands-on
I've tried copying the hive-site.xml and jar's into the sql-client and into the jobmanager, when I then execute :
Copy code
CREATE CATALOG c_hive WITH (
      'type' = 'hive',
      'hive-conf-dir' = './conf/');
with the updated/fully qualified hive-conf-dir specified I get either of the below 2 error's. this is all in docker containers spun up via a docker-compose file. i copied my jars into a subdir './hive/' where I found the native supplied jar's
Copy code
Flink SQL> CREATE CATALOG c_hive WITH (
>       'type' = 'hive',
>       'hive-conf-dir' = '/opt/flink/conf/');
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.catalog.exceptions.CatalogException: Failed to create Hive Metastore client
or
Copy code
Flink SQL> CREATE CATALOG c_hive WITH (
>       'type' = 'hive',
>       'hive-conf-dir' = '/opt/sql-client/conf/');
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.CatalogFactory' in the classpath.

Available factory identifiers are:

generic_in_memory
jdbc
r
which Docker Compose are you using?
one of the ones linked from that blog?
cos they should Just Work™
g
I've build up a docker-compose, that pulls some bits together might need to see if i can pull the bits from your docker-compose in rather. all attached.
...cos they should Just Work™.... and java does not compute together 😉
looking over your 2 blog posts... must be blind... not seeing the docker-compose am also scanning through your learn flink git repo https://github.com/decodableco/examples/tree/main/flink-learn
you will see i'm using the cnflient images. let me ask first, into which container are the hive file and the jar's suppose to go. understand easy out would be to simply use your decodeable images, but i'd rather understand and build... get it working, then i know what where how...
g
thanks... ok i was a couple of continents away from getting this working looking at what you have in there 😉
... trying to pull your images...
Copy code
failed to solve: hms-standalone: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
I'm def logged into docker
decided to cheat and copying your docker-compose bits into mine...
wanna get persistance so i don't have to redo things over and over... then back to what i'm actually trying to get working. 😉
anyone, how do i fix this.
Copy code
=> [flink  5/12] RUN sed -i "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: flink-jobmanager/g" ./conf/flink-conf.yaml                                                                                                                                 0.1s
 => ERROR [flink  6/12] RUN apt-get update &&     apt-get install -y vim lnav                                                                                                                                                                                        3.5s
------
 > [flink  6/12] RUN apt-get update &&     apt-get install -y vim lnav:
0.819 Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
0.819 Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
2.115 Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
2.130 Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
2.130   At least one invalid signature was encountered.
2.430 Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
2.553 Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
2.553   At least one invalid signature was encountered.
2.968 Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
2.968   At least one invalid signature was encountered.
3.382 Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
3.382   At least one invalid signature was encountered.
3.422 Reading package lists...
3.482 W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: At least one invalid signature was encountered.
3.482 E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
3.482 W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: At least one invalid signature was encountered.
3.482 E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
3.482 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: At least one invalid signature was encountered.
3.482 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
3.482 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: At least one invalid signature was encountered.
3.482 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
------
failed to solve: process "/bin/sh -c apt-get update &&     apt-get install -y vim lnav" did not complete successfully: exit code: 100