https://datahubproject.io logo
Join Slack
Powered by
hi guys, i tried to ingest some data from bigquery...
# ingestion
f
hi guys, i tried to ingest some data from bigquery i got these
Copy code
/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py:535: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
any ideea? 😄
b
What does 'datahub check plugins' say
At the cli
f
Copy code
Sources:
athena         (disabled)
azure-ad
bigquery       (disabled)
bigquery-usage (disabled)
data-lake      (disabled)
datahub-business-glossary
dbt
druid          (disabled)
elasticsearch  (disabled)
feast
file
glue           (disabled)
hive           (disabled)
kafka          (disabled)
kafka-connect  (disabled)
ldap           (disabled)
looker         (disabled)
lookml         (disabled)
mariadb        (disabled)
metabase       (disabled)
mode           (disabled)
mongodb        (disabled)
mssql          (disabled)
mysql          (disabled)
nifi
okta           (disabled)
openapi
oracle         (disabled)
postgres       (disabled)
redash         (disabled)
redshift       (disabled)
redshift-usage (disabled)
sagemaker      (disabled)
snowflake      (disabled)
snowflake-usage (disabled)
sqlalchemy     (disabled)
starburst-trino-usage (disabled)
superset       (disabled)
trino          (disabled)

Sinks:
console
datahub-kafka  (disabled)
datahub-rest
file

Transformers:
add_dataset_ownership
add_dataset_properties
add_dataset_tags
add_dataset_terms
mark_dataset_status
pattern_add_dataset_ownership
pattern_add_dataset_tags
pattern_add_dataset_terms
set_dataset_browse_path
simple_add_dataset_ownership
simple_add_dataset_properties
simple_add_dataset_tags
simple_add_dataset_terms
simple_remove_dataset_ownership
i use a docker image with these libraryes
Copy code
RUN python3 -m pip install --upgrade acryl-datahub
RUN python3 -m pip install --upgrade acryl-datahub[bigquery]
RUN python3 -m pip install --upgrade acryl-datahub[datahub-rest]
RUN python3 -m pip install --upgrade google-cloud-bigquery
RUN python3 -m pip install --upgrade google-cloud-storage
RUN python3 -m pip install --upgrade requests
s
As you can see from the above output these are disabled. So something is wrong in the docker image that you are using
Copy code
bigquery       (disabled)
bigquery-usage (disabled)
Cn you try these below commands?
Copy code
datahub check plugins
Copy code
python3 -m datahub check plugins
Copy code
python -c "import platform; print(platform.platform())"
python -c "import sys; print(sys.version); print(sys.executable); import datahub; print(datahub.__file__)"
python3 -c "import sys; print(sys.version); print(sys.executable); import datahub; print(datahub.__file__)"
Copy code
pip -vvv install 'acryl-datahub[bigquery]'
Copy code
pip freeze
And give output of these commands?
f
Copy code
acryl-datahub==0.8.23.1
altair==4.2.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
attrs==21.4.0
avro==1.11.0
avro-gen3==0.7.1
backcall==0.2.0
black==21.12b0
bleach==4.1.0
cachetools==4.2.4
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.10
click==8.0.3
click-default-group==1.2.2
debugpy==1.5.1
decorator==5.1.1
defusedxml==0.7.1
docker==5.0.3
entrypoints==0.3
executing==0.8.2
expandvars==0.7.0
future==0.18.2
google-api-core==2.4.0
google-auth==2.3.3
google-cloud-appengine-logging==1.1.0
google-cloud-audit-log==0.2.0
google-cloud-bigquery==2.32.0
google-cloud-core==2.2.2
google-cloud-logging==2.7.0
google-cloud-storage==2.1.0
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.54.0
great-expectations==0.14.1
greenlet==1.1.2
grpc-google-iam-v1==0.12.3
grpcio==1.43.0
grpcio-status==1.43.0
idna==3.3
importlib-metadata==4.10.1
ipykernel==6.7.0
ipython==8.0.1
ipython-genutils==0.2.0
ipywidgets==7.6.5
jedi==0.18.1
Jinja2==3.0.3
jsonpatch==1.32
jsonpointer==2.2
jsonschema==4.4.0
jupyter-client==7.1.1
jupyter-core==4.9.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mistune==0.8.4
more-itertools==8.12.0
mypy-extensions==0.4.3
nbclient==0.5.10
nbconvert==6.4.0
nbformat==5.1.3
nest-asyncio==1.5.4
notebook==6.4.7
numpy==1.22.1
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
platformdirs==2.4.1
progressbar2==4.0.0
prometheus-client==0.12.0
prompt-toolkit==3.0.24
proto-plus==1.19.8
protobuf==3.19.3
ptyprocess==0.7.0
pure-eval==0.2.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybigquery==0.10.2
pycparser==2.21
pydantic==1.9.0
Pygments==2.11.2
pyparsing==2.4.7
pyrsistent==0.18.1
python-dateutil==2.8.2
python-utils==3.1.0
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0
pyzmq==22.3.0
requests==2.27.1
rsa==4.8
ruamel.yaml==0.17.17
ruamel.yaml.clib==0.2.6
scipy==1.7.3
Send2Trash==1.8.0
six==1.16.0
SQLAlchemy==1.3.24
stack-data==0.1.4
stackprinter==0.2.5
tabulate==0.8.9
termcolor==1.1.0
terminado==0.12.1
testpath==0.5.0
toml==0.10.2
tomli==1.2.3
toolz==0.11.2
tornado==6.1
tqdm==4.62.3
traitlets==5.1.1
typing-extensions==3.10.0.2
typing-inspect==0.7.1
tzdata==2021.5
tzlocal==4.1
urllib3==1.26.8
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.2.3
widgetsnbextension==3.5.2
zipp==3.7.0
Open in Slack
PreviousNext
Powered by