prehistoric-lamp-76173
03/14/2024, 7:56 AMprehistoric-lamp-76173
03/14/2024, 9:13 AM<https://github.com/datahub-project/datahub/tree/master|master>
by anshbansal
<https://github.com/datahub-project/datahub/commit/0f2b15c93e3243ca37ca66b70465a43ab73bc71c|0f2b15c9>
- fix(ui/lineage): show data is too large error when limitation exceeds (#10038)
datahub-project/datahubprehistoric-lamp-76173
03/14/2024, 9:29 AMprehistoric-lamp-76173
03/14/2024, 10:29 AMprehistoric-lamp-76173
03/14/2024, 11:45 AMprehistoric-lamp-76173
03/14/2024, 12:50 PMprehistoric-lamp-76173
03/14/2024, 1:36 PMprehistoric-lamp-76173
03/14/2024, 1:52 PMprehistoric-lamp-76173
03/14/2024, 3:04 PMprehistoric-lamp-76173
03/14/2024, 3:19 PMtext
field in SYS_QUERY_TEXT
table can have at most 4k characters and then is split into chunks ordered by sequence
we can allow at most 16 (16*4 = 64) chunks to be included.
datahub-project/datahub
GitHub Actions: Run Smoke Tests (no_cypress_suite1)
GitHub Actions: Run Smoke Tests (no_cypress_suite0)
GitHub Actions: [Monitoring] Scan MCE consumer images for vulnerabilities
GitHub Actions: [Monitoring] Scan MAE consumer images for vulnerabilities
GitHub Actions: [Monitoring] Scan DataHub Upgrade images for vulnerabilities
GitHub Actions: [Monitoring] Scan GMS images for vulnerabilities
GitHub Actions: [Monitoring] Scan Frontend images for vulnerabilities
GitHub Actions: Build and Push DataHub MySQL Setup Docker Image
GitHub Actions: Build and Push DataHub MCE Consumer Docker Image
GitHub Actions: Build and Push DataHub MAE Consumer Docker Image
GitHub Actions: Build and Push DataHub Kafka Setup Docker Image
GitHub Actions: Build and Push DataHub GMS Docker Image
GitHub Actions: Build and Push DataHub Frontend Docker Image
GitHub Actions: Build and Push DataHub Upgrade Docker Image
GitHub Actions: Build and Push DataHub Elasticsearch Setup Docker Image
GitHub Actions: build (frontend, America/New_York)
GitHub Actions: build (frontend, UTC)
GitHub Actions: quickstart-compose-validation
✅ 12 other checks have passed
12/30 successful checksprehistoric-lamp-76173
03/14/2024, 3:36 PMprehistoric-lamp-76173
03/14/2024, 4:40 PMprehistoric-lamp-76173
03/14/2024, 4:56 PMPartitionKey["Type"]
So making the connector fail safe in case PartitionKey["Type"] not found for a table.
Also adding a debug logger, which will help to debug any issue when connector failed to process data for a particular table.
Checklist
☐ The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
☐ Links to related issues (if applicable)
☐ Tests for the changes have been added/updated (if applicable)
☐ Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
☐ For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub
datahub-project/datahubprehistoric-lamp-76173
03/14/2024, 5:01 PMprehistoric-lamp-76173
03/14/2024, 5:16 PMprehistoric-lamp-76173
03/14/2024, 5:36 PMprehistoric-lamp-76173
03/14/2024, 5:53 PMprehistoric-lamp-76173
03/14/2024, 6:43 PMprehistoric-lamp-76173
03/14/2024, 6:59 PMprehistoric-lamp-76173
03/14/2024, 7:16 PMprehistoric-lamp-76173
03/14/2024, 7:26 PMprehistoric-lamp-76173
03/14/2024, 7:32 PMprehistoric-lamp-76173
03/14/2024, 7:43 PMprehistoric-lamp-76173
03/14/2024, 8:35 PMprehistoric-lamp-76173
03/14/2024, 8:51 PMprehistoric-lamp-76173
03/14/2024, 9:07 PMprehistoric-lamp-76173
03/14/2024, 9:59 PMprehistoric-lamp-76173
03/14/2024, 10:43 PMprehistoric-lamp-76173
03/14/2024, 10:51 PMtype: hive
2. Click Save & Run
3. Get an error about required type
field
4. Update the recipe with type: hive
5. Click Save & Run
button
6. See the errors (logs are below):
1. ModuleNotFoundError: No module named 'thrift'
2. pyiceberg.exceptions.NotInstalledError: Apache Hive support not installed: pip install 'pyiceberg[hive]'
Expected behavior
All pypi packages 'pyiceberg[hive]' thrift
should be installed properly
Solution
Execute pip install
every time before execution of recipe
Screenshots
image
Desktop (please complete the following information):
• OS: MacOS Sonoma arm64
• Browser Chrome
• Version 122.0.6261.112
Additional context
1. Recipe:
source:
type: iceberg
config:
env: PROD
catalog:
name: iceberg-catalog
type: hive
config:
uri: '<https://hostname1:9083>'
s3.endpoint: '<https://hostname2>'
s3.access-key-id: '${secret1}'
s3.secret-access-key: '${secret2}'
table_pattern:
allow:
- 'test.*'
profiling:
enabled: false
3. Error logs: exec-urn_li_dataHubExecutionRequest_1d2b870e-81e9-477a-8869-39505a9f2b3d.log
4. Even adding Extra Pip Libraries does not help
Extra Pip Libraries9 9. Datahub version 0.12.1 10. As I see, it is not fixed in 0.13.0 from 0.12.1 https://github.com/datahub-project/datahub/commits/v0.12.1/metadata-ingestion/src/datahub/ingestion/source/iceberg
datahub-project/datahubprehistoric-lamp-76173
03/14/2024, 11:08 PM