Hi Team, I'm using Flink SQL to sync from Mongo to...
# troubleshooting
y
Hi Team, I'm using Flink SQL to sync from Mongo to Hive. but the Mongo collection has 2.8 billion+ data, how can i spend less time to sync data?
env: Mongo: 6.0.6 (master/slave) Flink: 1.16.2 (standalone) (16c * 32g) Hive: 3.1.0 my operations like this:
Copy code
sql-client.sh -i /data/flink/init.sql
create table:
Copy code
-- create mongo table
CREATE TABLE flink_database.md_xxxbaseinfo (
  `_id`           String,
  `str`           String,
  `time`          TIMESTAMP_LTZ,
  `arr1`          array<ROW<`inv` String , `detail` ROW<`subSum` string , `aubSum` string , `subDetails` array<Map<String ,String >> , `aubDetails` array<Map<String ,String >> > , `invType_CN` String  >> ,
  `int`            INT
) WITH (
   'connector' = 'mongodb',
   'uri' = '<mongodb://root:mingyang100@xxx:27017>,xxx:27017,xxx:27017/admin?replicaSet=rs0&connectTimeoutMS=10000&authSource=admin',
   'database' = 'test',
   'collection' = 'xxxBaseInfo'
);

-- create hive table
SET table.sql-dialect=hive;

CREATE external TABLE ods.ods_md_xxx_xxxBaseInfo(
  `_id` string,
  `str` string, 
  `time` TIMESTAMP, 
  `arr1` array<STRUCT<`inv`:String, `detail`:STRUCT<`subSum`:String, `aubSum`:String, `subDetails`: array<Map<String,String>>, `aubDetails`: array<Map<String,String>>>, `invType_CN`: String >>,
  `int` int
  )
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde' 
WITH SERDEPROPERTIES ( 
  'collection.delim'=',', 
  'field.delim'='\t', 
  'serialization.format'='\t') 
STORED AS ORC
LOCATION
  '<hdfs://xxx:8020/warehouse/tablespace/external/hive/ods.db/mongo/xxx/xxxBaseInfo>';

-- sync data
SET table.sql-dialect=default;
SET 'table.local-time-zone' = 'UTC';

INSERT INTO ods.ods_md_xxx_xxxBaseInfo
 SELECT *
FROM flink_database.md_xxxbaseinfo;
a
What is the parallelism of your Flink job?
Regardless, the idea is to try to read the data in parallel and to play around with different partition strategies (depends on your data and how it is stored in Mongo) https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/mongodb/#partitioned-scan
y
@Alexander Fedulov Thanks for your reply. My Flink SQL config just use default config, and my Mongo cluster is
master/slave
. I have read the relevant official documents,
'scan.partition.strategy'='default'
, But i still don't know how to improve job parallelism? My Flink Job and My machine resource usage:
@Martijn Visser Hello👋🏻, Can you help me look at this problem? Thank you, very much.
m
Please follow the rules and don't @ people
y
Oh, I'm Sorry