Hey guys I tried to use the new feature OpenApi ingestion, but seems like did something wrong. I set URL and swagger_file and run ingestion, but saw Unknown error for reaching endpoint I see that ingestor tries to do GET request without any get params. As the result my API returns 400 code and ingestors fails. What do I do wrong?

@stale-jewelry-2440 might be able to help here.

Hi @lively-jackal-83760, can you please show me your config file?

@stale-jewelry-2440 Hi

source:
  type: openapi
  config:
    name: test_endpoint
    url: <http://my-host:8080/>
    swagger_file: v3/api-docs/rt-kpi-api-v1 

sink:
  type: datahub-rest
  config:
    server: <http://localhost:8080>

and what swagger returns

{
  "openapi": "3.0.1",
  "info": {
    "title": "title",
    "version": "1.0.8"
  },
  "servers": [
    {
      "url": "<http://localhost:8080>",
      "description": "Generated server url"
    }
  ],
  "paths": {
    "/v1/rt-kpi": {
      "get": {
        "tags": [
          "dashboard-controller"
        ],
        "operationId": "get",
        "parameters": [
          {
            "name": "metrics",
            "in": "query",
            "required": true,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "string",
                "enum": [
                  "revenue",
                  "revenue_cumulative",
                  "ftd",
                  "installs",
                  "dau",
                  "dau_cumulative"
                ]
              }
            }
          },
          {
            "name": "timePeriod",
            "in": "query",
            "required": true,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "string",
                "enum": [
                  "today",
                  "yesterday",
                  "last_week_same_day",
                  "avg_last_four_weeks_same_day",
                  "previous_month",
                  "avg_last_four_month",
                  "daily_avg"
                ]
              }
            }
          },
          {
            "name": "startTimestamp",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string",
              "format": "date-time"
            },
            "example": "2021-09-01T00:00:00.587Z"
          },
          {
            "name": "endTimestamp",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string",
              "format": "date-time"
            },
            "example": "2021-09-02T00:00:00.587Z"
          },
          {
            "name": "country",
            "in": "query",
            "required": false,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "integer",
                "format": "int32"
              }
            }
          },
          {
            "name": "platform",
            "in": "query",
            "required": false,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "integer",
                "format": "int32"
              }
            }
          },
          {
            "name": "division",
            "in": "query",
            "required": false,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "integer",
                "format": "int32"
              }
            }
          },
          {
            "name": "game",
            "in": "query",
            "required": false,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "integer",
                "format": "int32"
              }
            }
          },
          {
            "name": "splitBy",
            "in": "query",
            "required": false,
            "schema": {
              "uniqueItems": true,
              "type": "array",
              "items": {
                "type": "string",
                "enum": [
                  "app",
                  "game",
                  "studio",
                  "division"
                ]
              }
            }
          },
          {
            "name": "aggregateBy",
            "in": "query",
            "required": true,
            "schema": {
              "type": "integer",
              "format": "int32"
            },
            "example": 3600
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "*/*": {
                "schema": {
                  "$ref": "#/components/schemas/KpiResponse"
                }
              }
            }
          },
          "400": {
            "description": "Bad Request",
            "content": {
              "*/*": {
                "schema": {
                  "$ref": "#/components/schemas/ValidationErrorResponse"
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "ValidationErrorResponse": {
        "type": "object",
        "properties": {
          "violations": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/Violation"
            }
          }
        }
      },
      "Violation": {
        "type": "object",
        "properties": {
          "fieldName": {
            "type": "string"
          },
          "message": {
            "type": "string"
          }
        }
      },
      "GroupTag": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string"
          },
          "value": {
            "type": "string"
          }
        }
      },
      "KpiDataPoint": {
        "type": "object",
        "properties": {
          "datetime": {
            "type": "integer",
            "format": "int64"
          },
          "value": {
            "type": "number",
            "format": "double"
          }
        }
      },
      "KpiRequest": {
        "required": [
          "metrics",
          "timePeriod"
        ],
        "type": "object",
        "properties": {
          "metrics": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "string",
              "enum": [
                "revenue",
                "revenue_cumulative",
                "ftd",
                "installs",
                "dau",
                "dau_cumulative"
              ]
            }
          },
          "timePeriod": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "string",
              "enum": [
                "today",
                "yesterday",
                "last_week_same_day",
                "avg_last_four_weeks_same_day",
                "previous_month",
                "avg_last_four_month",
                "daily_avg"
              ]
            }
          },
          "startTimestamp": {
            "type": "string",
            "format": "date-time"
          },
          "endTimestamp": {
            "type": "string",
            "format": "date-time"
          },
          "country": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "integer",
              "format": "int32"
            }
          },
          "platform": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "integer",
              "format": "int32"
            }
          },
          "division": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "integer",
              "format": "int32"
            }
          },
          "game": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "integer",
              "format": "int32"
            }
          },
          "splitBy": {
            "uniqueItems": true,
            "type": "array",
            "items": {
              "type": "string",
              "enum": [
                "app",
                "game",
                "studio",
                "division"
              ]
            }
          },
          "aggregateBy": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "KpiResponse": {
        "required": [
          "query",
          "result"
        ],
        "type": "object",
        "properties": {
          "query": {
            "$ref": "#/components/schemas/KpiRequest"
          },
          "result": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/KpiResult"
            }
          }
        }
      },
      "KpiResult": {
        "type": "object",
        "properties": {
          "groupTags": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/GroupTag"
            }
          },
          "dataPoints": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/KpiDataPoint"
            }
          }
        }
      }
    }
  }
}

@lively-jackal-83760 @stale-jewelry-2440 - I am trying to ingest OpenAPI using this sample https://petstore.swagger.io I am getting

2022-02-21 16:48:13,001] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/ec2-user/.venvs/airflow/lib/python3.7/site-packages/datahub/ingestion/source/openapi_parser.py", line 124, in get_endpoints
    base_res = p_o["get"]["responses"]["200"]
KeyError: '200'

During handling of the above exception, another exception occurred:

Any suggestions on where I might be going wrong ?

the error is relative to the

/user/logout

endpoint, which has an unforeseen shape. Unfortunately, adding it to the list of ignored endpoints does not help, since that specific check happens later. Waiting for the fix to be deployed, I can suggest you an hot-fix in private. In any case, the correct ingestion file configuration for this example is:

source:
  type: openapi
  config:
    name: petstore
    url: <https://petstore.swagger.io/>  # the v2 goes below
    swagger_file: v2/swagger.json
    forced_examples:
      /pet/{petId} : "1"  # should be a string

Hey there! 👋 Make sure your message includes the following information if relevant, so we can help more effectively! 1. Which DataHub version are you using? (e.g. 0.12.0) 2. Please post any relevant error logs on the thread!