https://datahubproject.io logo
Join Slack
Powered by
I get the following error: ```"No root resource d...
# ingestion
c
I get the following error:
Copy code
"No root resource defined for path '/entities'","status":404}(base)
If I try to ingest via curl. The other option via acryl addon works...
Copy code
curl '<http://localhost:8080/entities?action=ingest>' -X POST --data '{
   "entity":{
      "value":{
         "com.linkedin.metadata.snapshot.DatasetSnapshot":{
            "aspects":[
               {
                  "com.linkedin.common.Ownership":{
                     "owners":[
                        {
                           "owner":"urn:li:corpuser:fbar",
                           "type":"DATAOWNER"
                        }
                     ],
                     "lastModified":{
                        "time":0,
                        "actor":"urn:li:corpuser:fbar"
                     }
                  }
               },
               {
                  "com.linkedin.common.InstitutionalMemory":{
                     "elements":[
                        {
                           "url":"<https://www.linkedin.com>",
                           "description":"Sample doc",
                           "createStamp":{
                              "time":0,
                              "actor":"urn:li:corpuser:fbar"
                           }
                        }
                     ]
                  }
               },
               {
                  "com.linkedin.schema.SchemaMetadata":{
                     "schemaName":"FooEvent",
                     "platform":"urn:li:dataPlatform:foo",
                     "version":0,
                     "created":{
                        "time":0,
                        "actor":"urn:li:corpuser:fbar"
                     },
                     "lastModified":{
                        "time":0,
                        "actor":"urn:li:corpuser:fbar"
                     },
                     "hash":"",
                     "platformSchema":{
                        "com.linkedin.schema.KafkaSchema":{
                           "documentSchema":"{\"type\":\"record\",\"name\":\"MetadataChangeEvent\",\"namespace\":\"com.linkedin.mxe\",\"doc\":\"Kafka event for proposing a metadata change for an entity.\",\"fields\":[{\"name\":\"auditHeader\",\"type\":{\"type\":\"record\",\"name\":\"KafkaAuditHeader\",\"namespace\":\"com.linkedin.avro2pegasus.events\",\"doc\":\"Header\"}}]}"
                        }
                     },
                     "fields":[
                        {
                           "fieldPath":"foo",
                           "description":"Bar",
                           "nativeDataType":"string",
                           "type":{
                              "type":{
                                 "com.linkedin.schema.StringType":{

                                 }
                              }
                           }
                        }
                     ]
                  }
               }
            ],
            "urn":"urn:li:dataset:(urn:li:dataPlatform:foo,bar,PROD)"
         }
      }
   }
}'
This is probably also the reason why I can't emit lineage from airflow
g
I tried that curl command and it worked for me. Can you share more details about your deployment? How are you running datahub? Are you sure GMS is running at port 8080?
c
I still run it with docker-compose. Dockers shows me the following. It seems to me that GMS is running at port 8080:
Copy code
d05d8157486f   linkedin/datahub-mce-consumer:latest     "/bin/sh -c /datahub…"   2 months ago   Up 2 months (healthy)   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                                                        datahub-mce-consumer
e0da851d2bac   linkedin/datahub-frontend-react:latest   "datahub-frontend/bi…"   2 months ago   Up 2 months (healthy)   0.0.0.0:9002->9002/tcp, :::9002->9002/tcp                                                        datahub-frontend-react
8197e6e527cd   linkedin/datahub-mae-consumer:latest     "/bin/sh -c /datahub…"   2 months ago   Up 2 months (healthy)   9090/tcp, 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp                                              datahub-mae-consumer
908129f26597   linkedin/datahub-gms:latest              "/bin/sh -c /datahub…"   2 months ago   Up 2 months (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                        datahub-gms
7b6d01c49b22   neo4j:4.0.6                              "/sbin/tini -g -- /d…"   2 months ago   Up 2 months             0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp   neo4j
c66301c89516   mysql:5.7                                "docker-entrypoint.s…"   2 months ago   Up 2 months             0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp                                             mysql
6b44594de535   confluentinc/cp-kafka-rest:5.4.0         "/etc/confluent/dock…"   3 months ago   Up 2 months             0.0.0.0:8082->8082/tcp, :::8082->8082/tcp                                                        kafka-rest-proxy
58f33f937cdc   confluentinc/cp-schema-registry:5.4.0    "/etc/confluent/dock…"   3 months ago   Up 2 months             0.0.0.0:8081->8081/tcp, :::8081->8081/tcp                                                        schema-registry
62054597f1cd   confluentinc/cp-kafka:5.4.0              "/etc/confluent/dock…"   3 months ago   Up 2 months             0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp         broker
73c8620855c8   kibana:7.9.3                             "/usr/local/bin/dumb…"   3 months ago   Up 2 months             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                        kibana
dc6fccbf69e6   elasticsearch:7.9.3                      "/tini -- /usr/local…"   3 months ago   Up 2 months (healthy)   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp                                              elasticsearch
5d80304cacb2   confluentinc/cp-zookeeper:5.4.0          "/etc/confluent/dock…"   3 months ago   Up 2 months             2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp                                    zookeeper
I mean my recipe via the acryl-ingestion add-in works and there I define as sink the kafka rest api on 8080. But maybe I should set it up without docker?
g
whats the output of 
curl localhost:8080/config
?
c
Copy code
{"exceptionClass":"com.linkedin.restli.server.RestLiServiceException","stackTrace":"com.linkedin.restli.server.RestLiServiceException [HTTP Status:404]: No root resource defined for path '/config'\n\tat com.linkedin.restli.server.RestLiServiceException.fromThrowable(RestLiServiceException.java:315)\n\tat com.linkedin.restli.server.BaseRestLiServer.buildPreRoutingError(BaseRestLiServer.java:158)\n\tat com.linkedin.restli.server.RestRestLiServer.buildPreRoutingRestException(RestRestLiServer.java:203)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:177)\n\tat com.linkedin.restli.server.RestRestLiServer.doHandleRequest(RestRestLiServer.java:164)\n\tat com.linkedin.restli.server.RestRestLiServer.handleRequest(RestRestLiServer.java:120)\n\tat com.linkedin.restli.server.RestLiServer.handleRequest(RestLiServer.java:132)\n\tat com.linkedin.restli.server.DelegatingTransportDispatcher.handleRestRequest(DelegatingTransportDispatcher.java:70)\n\tat com.linkedin.r2.filter.transport.DispatcherRequestFilter.onRestRequest(DispatcherRequestFilter.java:70)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:72)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.TimedNextFilter.onRequest(TimedNextFilter.java:55)\n\tat com.linkedin.r2.filter.transport.ServerQueryTunnelFilter.onRestRequest(ServerQueryTunnelFilter.java:58)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:72)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.TimedNextFilter.onRequest(TimedNextFilter.java:55)\n\tat com.linkedin.r2.filter.message.rest.RestFilter.onRestRequest(RestFilter.java:50)\n\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:72)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\n\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\n\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\n\tat com.linkedin.r2.filter.FilterChainImpl.onRestRequest(FilterChainImpl.java:96)\n\tat com.linkedin.r2.filter.transport.FilterChainDispatcher.handleRestRequest(FilterChainDispatcher.java:75)\n\tat com.linkedin.r2.util.finalizer.RequestFinalizerDispatcher.handleRestRequest(RequestFinalizerDispatcher.java:61)\n\tat com.linkedin.r2.transport.http.server.HttpDispatcher.handleRequest(HttpDispatcher.java:101)\n\tat com.linkedin.r2.transport.http.server.AbstractR2Servlet.service(AbstractR2Servlet.java:105)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat com.linkedin.restli.server.spring.ParallelRestliHttpRequestHandler.handleRequest(ParallelRestliHttpRequestHandler.java:61)\n\tat org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:73)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:536)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:494)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.linkedin.restli.server.RoutingException: No root resource defined for path '/config'\n\tat com.linkedin.restli.internal.server.RestLiRouter.process(RestLiRouter.java:139)\n\tat com.linkedin.restli.server.BaseRestLiServer.getRoutingResult(BaseRestLiServer.java:139)\n\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:173)\n\t... 62 more\n","message":"No root resource defined for path '/config'","status":404}/
g
I see- it looks like your GMS is running on an older version!
Try upgrading GMS to the latest version
c
Yes, I had this thought now as well. Now I'm getting a different error:
Copy code
[2021-07-27 15:16:14,027] {datahub.py:122} ERROR - ('Unable to emit metadata to DataHub GMS', {'message': "HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /entities?action=ingest (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4d289c5128>: Failed to establish a new connection: [Errno 111] Connection refused',))"})
Any thoughts on that? Thanks again.
I've upgraded both acryl-datahub and datahub itself
I can ingest for example
Copy code
mce-files/bootstrap_mce.json
without error
g
Nice! Glad ingesting the bootstrap data is working for you.
Are you running ingestion and curling from the same terminal? Maybe one is ssh'd into a machine while another is not?
c
Mhh I think there is a problem with the port mapping since I've tweaked airflow to :8083 that was pointing to :8080 before, but if I do localhost:8080 maybe it is somehow pointing to airflow internally? I have a domain/ip if I use this insted +:8080 than it works and actually shows in datahub
Nice
localhost would be nicer though, better to look into that further
g
ok- are you unblocked then?
c
Yes for the moment I am. Thanks for your help. Much appreciated
g
Great!
Open in Slack
PreviousNext
Powered by