https://datahubproject.io logo
Join Slack
Powered by
Hi, does anyone know how to define credentials in ...
# ingestion
n
Hi, does anyone know how to define credentials in recipe file or handle permission error when ingesting from 
bigquery-usage
? ā€¢ 
options.credentials_path
, or 
extra_client_options.credentials_path
does not work ( fails to run the file -> 
got an unexpected keyword arguement
or 
extra fields not permitted
) ā€¢ I tried 
export GOOGLE_APPLICATION_CREDENTIALS
-> file runs but it stops with the error : 
the caller does not have permission
ā€¢ bigquery ingestion under same environment & configs works without errors.
here's my recipe ....
Copy code
source:
  type: bigquery-usage
  config:
    projects:
      - <my_project_name>
    top_n_queries: 10
#   extra_client_options: # this fails
#     credentials_path: <path_to_file>
  

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"
here's the error message when i export the credentials
Copy code
Forbidden: 403 POST <https://logging.googleapis.com/v2/entries:list?prettyPrint=false>: The caller does not have permission
m
Are you using a service account?
b
stupid question, but does the service account have enough permissions for the usage? šŸ˜… they are different to just bigquery (meta-)data ingestion
n
I am using service account, but i'll check the permissions. Thanks!
m
n
Hi, i adjusted IAMs of the service account and it worked, thanks! But i still got questions.. ā€¢ Is there a way to define 
credentials_path
in bigquery-usage recipe? ā€¢ Is it possible for bigquery-usage to not write any records ? ā—¦ when I set start_time ~ end_time to a be couple of hours, 
records_written
is 0 (although it's during workhour) ā—¦ when I set start_time ~ end_time to be one day, the process takes a loooong time, (so i just stopped bc i was afraid of the cost ...)
it returns something like this.
šŸ‘€ 1
4 Views
Open in Slack
PreviousNext
Powered by