Slackbot
10/09/2023, 5:39 AMRajnesh
10/09/2023, 9:29 AMRajnesh
10/09/2023, 8:08 PMAlexander Preuß
10/11/2023, 2:04 PMHang Chen
10/11/2023, 3:30 PMHang Chen
10/11/2023, 3:48 PMHang Chen
10/11/2023, 3:49 PMRajnesh
10/11/2023, 4:19 PMRajnesh
10/11/2023, 4:30 PMfrom pulsar.schema import *
import pulsar
import time
from dataclasses import dataclass
from typing import List
import uuid
from datetime import datetime
def post_data():
class AuditEvent(Record):
event = String()
description = String()
affected_id= String()
affected_type= String()
actor_id=String()
actor_type=String()
print("schema creation in process")
client = pulsar.Client('<pulsar://localhost:6650>')
time.sleep(5)
producer = client.create_producer(
topic='public/default/audit_file',
schema=JsonSchema(AuditEvent),
)
print("post_data")
producer.send(AuditEvent(
event='audit',
description='Creating Event',
affected_id='328006c1-793f-43be-93824906',
affected_type='ACTOR_TYPE_SERVICE',
actor_id='018ac9c5-f594-722f-aeda7524b7a',
actor_type='ACTOR_TYPE_SERVICE'
)
)
print("data posted")
post_data()Rajnesh
10/11/2023, 4:34 PMHang Chen
10/12/2023, 1:26 AMRajnesh
10/12/2023, 6:05 AMRajnesh
10/13/2023, 7:42 AMRajnesh
10/19/2023, 7:34 AMHang Chen
10/19/2023, 7:45 AMHang Chen
10/19/2023, 7:46 AMHang Chen
10/20/2023, 3:59 AMYan Zhao
10/20/2023, 9:34 AMRajnesh
10/25/2023, 8:01 PMYan Zhao
10/30/2023, 8:59 AMYan Zhao
10/30/2023, 5:21 PM120s , the delta writer commit interval is 120s. After commit, the parquet file will be flushed to file system.Rajnesh
10/30/2023, 7:18 PMYan Zhao
10/31/2023, 1:54 AMmaxRecordsPerCommit , if so, commit.
2. Whether the time between the current time and the last commit exceeds maxCommitInterval , if so, commit.
So I guess that the parquet file haven’t commit yet, you can start the sink writer, and continuously sending messages to the topic for more than 120 seconds. Then check the parquet file status.
If possible, could you attach the logs when you test it. Thanks.Hang Chen
10/31/2023, 3:03 AMmaxCommitInterval to 10Rajnesh
10/31/2023, 6:48 AMmaxCommitInterval to 10Yan Zhao
10/31/2023, 9:11 AM