https://datahubproject.io logo
Join Slack
Powered by
Hello, using docker-compose.quickstart.yml file fr...
# ingestion
h
Hello, using docker-compose.quickstart.yml file from git hub repo. I am able bring up all the components on aws ec2. Later I ingested the athena tables. Can someone please help me where the data will be stored in the ec2. I would like to take a backup of the data and push it to s3.
b
In a docker volume in the vm
I suppose you could exec into the MySQL vm and do a dump, then copy out the file from the container to the host
h
I checked the /var/lib/docker/volumes/datahub_mysql/_data it has some data. I copied the _data into efs and mounted to a new ec2 and data is not seen in new ec2 instance.
@better-orange-49102 any idea?
b
So u tried to copy out the file that represents a docker volume to another place... I'm not familiar enough with volumes to know if it would work.
h
Yep it has below list of files in the quickstart_mysql/_data folder
Copy code
auto.cnf    ca.pem           client-key.pem  ib_buffer_pool  ib_logfile0  ibtmp1  performance_schema  public_key.pem   server-key.pem
ca-key.pem  client-cert.pem  datahub         ibdata1         ib_logfile1  mysql   private_key.pem     server-cert.pem  sys
and I copied them to efs.
but no luck. 😞
s
Are you running mysql in docker container?
If yes, then connect using mysql client to your mysql db and do a mysqldump to take backup
If you are in AWS strongly recommend to use AWS RDS which supports automated backups
Copying that raw mysql file will not work unless you look into understanding how mysql stores the raw data. That is going to be more time consuming process. Strongly recommend using AWS RDS and use their automated backups
h
Thanks Aseem.. I will make use of AWS RDS MYSQL and install it.
Hi @square-activity-64562 I have created RDS and ingested the athena data to Mysql db and I created new machine and update the mysql to rds and ran 
docker-compose -f docker-compose.quickstart.yml up
. It doesn't show the ahtena data in UI. Do I need to create recipe for mysql and ingest the data to see the data in UI?
s
Is your server pointing to mysql? Did you change the quickstart to point to that?
h
Yeah @square-activity-64562 I changed the quickstart file to point mysql rds.
Open in Slack
PreviousNext
Powered by