This message was deleted.

Can you share the error here? Are you able to produce to Pulsar using a traditional client, e.g., the pulsar-client CLI tool? Also, if you have secured the cluster, be sure to grant the Databricks notebook user permissions to create topics.

Here is the error:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d62a23e6-2ac4-473a-9b47-5944c07a4142/lib/python3.10/site-packages/pulsar/__init__.py:743, in Client.create_producer(self, topic, producer_name, schema, initial_sequence_id, send_timeout_millis, compression_type, max_pending_messages, max_pending_messages_across_partitions, block_if_queue_full, batching_enabled, batching_max_messages, batching_max_allowed_size_in_bytes, batching_max_publish_delay_ms, chunking_enabled, message_routing_mode, lazy_start_partitioned_producers, properties, batching_type, encryption_key, crypto_key_reader, access_mode)
    740     raise ValueError("Batching and chunking of messages can't be enabled together.")
    742 p = Producer()
--> 743 p._producer = self._client.create_producer(topic, conf)
    744 p._schema = schema
    745 p._client = self._client

BrokerMetadataError: Pulsar error: BrokerMetadataError

Here is the relevant Notebook cell:

I think the issue is with the topic name itself. In Pulsar, topic names take the form of {tenant}/{namespace}/{topic-name}, e.g.

public/default/my-topic

, and from what I can see, you are using just

my-topic

which will lead to a metadata error because Pulsar cannot find a topic with that name

Thanks for the response! That's interesting - this code has worked outside this specific environment which is why I'm confused. I thought that

create_producer

would create the topic if it didn't exist

It looks like changing the topic name to include the namespace worked!