https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
Hello Daniel, This doc lists in detail the difference running workers are a separate process and running it as part of the broker
d
yes i did read that, asking what is the use of the worker exposed port šŸ™‚
b
Function workers when running with broker, still inherit settings the settings like authentication and configuration with the help of broker. The same applies even if itā€™s running without the broker.
So the function workers listen with a service url
And it listens on a port for the calls from the client application.
d
yes, i did configure the functions_worker.yml and the workers started as expected
but what is that port used for ? does the broker talk with the workers ?
so the port is only for client applicaton and not for brokers correct ?
b
Correct. Thatā€™s what Iā€™d assume looking at this diagram
d
i see, was not very clear to me
thank you for your time!
b
Ofcourse Daniel. No worries at all.
d
one more question, if the worker is standalone, means that it not not really have any interaction with the brokers, besides consuming from the topic and producing to another, correct ?
b
Yap. Mostly to get some basic config settings from the broker and depending on the business logic to read and write to topics
Main reasons people choose to separate workers out it to isolate the business processing from brokers to not cause any resource issues
d
yes, thats why we also chose this, we need isolation from broker resources
āœ… 1
but it was not clear for the DOCS how isolated from the brokers the workers are, i see at INIT they make a PUT call to public/functions, so thats why i was wondering if there is any other interaction with the brokers
thanks again for your time, much apprciated!
d
@Daniel Ciocirlan, All Pulsar functions interact directly with the Pulsar Brokers to consume and publish messages, so there is always that ā€œdependencyā€ between the two. However, where you choose to run the Pulsar Function instances is up to you, and dictates how ā€œisolatedā€ the workers are from the brokers. There are three runtime environments you can use for Functions, the first is ā€œthread-modeā€ which runs the Functions as threads inside the broker instances themselves. This option presents the opportunity for a bad function to crash an entire broker at worst, and compete for CPU and RAM resources at best. The next option is ā€œprocess-modeā€ and this is where a separate process is created for each Function instance, and the Function Workers are used to host these processes. This solves the issues of the previous approach by separating the Functions from the Brokers entirely.
When you are administering Pulsar Functions, you issue commands to the Pulsar Admin API, which is hosted on the broker. These commands are then forwarded to the Function workers over the port you asked about earlier. It is a communication channel between the two
d
Hey David nice to see you again šŸ™‚
šŸ‘‹ 1
when i was thinking for interaction i was thinking the brokers actually manage the functions on the workers
i can understand that the interaction of consuming and producing to the broker
i see the workers publish some metadata on the public/functions
and thats why i was wondering if they are managed by the brokers
Copy code
These commands are then forwarded to the Function workers over the port you asked about earlier. It is a communication channel between the two
this is very useful thank you!
so you manage the function from the broker and not hte standalone function workers
d
pulsar-admin function create ā€¦.
<-- goes to the broker. The broker looks at its runtime config and sees that it is ā€œprocess modeā€ so it forwards the instructions to the function workers to create the function instance, etc.
d
i asume you load the JAR from the brokers then
how does a broker discover the workers if they are on another machine ? when the functions owkers register to the ZK ?
d
Yes, the function artifacts are submitted to the broker, and written to BookKeeper for storage. The Function workers read the JARs off of BookKeeper.
āœ… 1
The function workers register with the brokers and form a ā€œresource poolā€ for sorts. The brokers need to know what resources are available on the worker nodes when they assign Functions to them and try to honor the functionā€™s resource requests.
d
ah cool
so using
Copy code
curl <broker-ip>:8080/admin/v2/worker/cluster
would display the standalone workers cluster
d
It is not as sophisticated as a full-fledge resource negotiator like Mesos, Yarn, or K8s scheduler but it works
I would assume so, but I donā€™t have the admin API committed to memory just yet. šŸ˜ƒ
b
Yap. Thatā€™s the command the doc says
This gives the workerid, hostname and port
d
pulsar-admin functions-worker get-cluster
so i am having this problem then
Copy code
nc -zv 10.66.221.117 6751
Ncat: Version 7.50 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.66.221.117:6751.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
this is from the broker to the worker
but the 
functions-worker get-cluster
times out
Copy code
functions-worker get-cluster
null

Reason: java.util.concurrent.TimeoutException
Copy code
[06/Feb/2023:16:44:54 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false&getEarliestTimeInBacklog=false HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.10.2" 2
Copy code
javax.servlet.ServletException: java.lang.UnsupportedOperationException: Pulsar Function Worker is not enabled, probably functionsWorkerEnabled is set to false
but it's set to false since i don't want to start it from the broker
d
I think you need to set the 
functionsWorkerEnabled
property to true regardless of how you plan on starting the workers.
d
oke i see, let me try that šŸ™‚
well if i do that
it tries to start the worker on the broker
Copy code
ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
i don't want it to start on the broker at all
d
Are you using the apache helm charts?
d
no, bare metal
šŸ¤” 1
so if i enable in broker.conf, it tries to start it locally as a process
looking here for admin commands
i have no functions loaded, only want to list the workers available to the broker
b
@David K for some reason, the docs explicitly say we need to set the 
functionsWorkerEnabled
property to false for starting the workers outside the brokers
Been a while, need to test this out myself
d
Based on the exception above, there appears to be an error in the code that assumes that this property needs to be set to ā€œtrueā€. @Daniel Ciocirlan what was the command you issued that generated this error ā€œjava.lang.UnsupportedOperationException: Pulsar Function Worker is not enabled, probably functionsWorkerEnabled is set to falseā€ ?
b
Cc: @Ming fyi
Iā€™ll try to also reproduce this today on my end when I find some time
@Ming do you recollect on top of your mind what value do we set for the function workers outside of the broker. Docs say it needs to be false for running outside of the brokers
d
@David K it's
Copy code
pulsar-admin functions-worker get-cluster
the workers are registered in ZK
si the broker should be able to see them
also port 6751 is open from the broker to the worker
and netstat shows the worker listening on 6751
d
Ok, so the issue is in the admin rest api call.
d
thanks for all the great help David, much appreciated
šŸ‘ 1
hey guys any ideas why the workers have this in the logs :
Copy code
Feb 07 13:29:59 pulsar-dev-va6-v0017-pulsarworker-i-094b5002dda8925d8 pulsar[17144]: 13:29:59.365 [function-web-25-6] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7c67ae80
mainly :
Copy code
connectString=localhost:2181
even if the Zk has defined IPs in function_woerks.yml ?
but at the start of the worker we have :
Copy code
INFO  org.apache.pulsar.functions.worker.WorkerUtils - initialize DistributedLog Namespace with ledgersStoreServers: 10.66.219.210:2181
the corect IP
due to this we get :
Copy code
ERROR org.apache.distributedlog.bk.SimpleLedgerAllocator - Error creating ledger for allocating /pulsar/functions/public/default/TokenGenerationInfoDeduplicationFunction/8c6f6ae9-7db0-4638-8f4c-28d082144dfe-pulsar-test-1.0-SNAPSHOT.jar/<default>/allocation :
i thinks it's from here :
Copy code
INFO  org.apache.distributedlog.impl.BKNamespaceDriver - Created shared client builder bk:<distributedlog://10.66.219.210:2181>,10.66.220.38:2181,10.66.221.102:2181,10.66.219.212:2181,10.66.220.50:2181/pulsar/functions:factory_writer_shared : zkServers = localhost:2181, ledgersPath = /ledgers, numIOThreads = 8
that :
Copy code
zkServers = localhost:2181
d
Is that entry in your 
function_worker.yaml
config file? What is the value of your 
configurationMetadataStoreUrl
property?
d
so after a lot of debugging we found the issue
it seems that init of the function workers writes bad data to ZK
we had this :
Copy code
1
BKDL
{"1":{"str":"localhost:2181"},"2":{"str":"/ledgers"},"3":{"tf":1},"4":{"tf":0},"5":{"str":"localhost:2181"},"6":{"str":"localhost:2181"},"7":{"str":"localhost:2181"}}
but the config file 
function_worker.yaml
was with the correct data
we had to manually update the Zk data to the correct IPs
the function loaded using the worker IP and PORT
because the broker has the problem we discussed above
so having another issue, the function worker does not seem to use the SSL params provided connecting to 6651 (the source topic), i can see in the Proxy : 
PEER_DID_NOT_RETURN_A_CERTIFICATE
the SSL param are set :
Copy code
#### tls configuration for worker service
# Enable TLS
tlsEnabled: true
# Path for the TLS certificate file
tlsCertificateFilePath: /opt/pulsar/ssl/certificate.pem
# Path for the TLS private key file
tlsKeyFilePath: /opt/pulsar/ssl/key.pem
# Path for the trusted TLS certificate file
tlsTrustCertsFilePath: /opt/pulsar/ssl/ca.pem
d
This is for communication between the function workers and the brokers, correct?
Copy code
useTLS: true
pulsarServiceUrl: <pulsar+ssl://localhost:6651/>
pulsarWebServiceUrl: <https://localhost:8443>

tlsEnabled: true
tlsCertificateFilePath: /path/to/functions-worker.cert.pem
tlsKeyFilePath:         /path/to/functions-worker.key-pk8.pem
tlsTrustCertsFilePath:  /path/to/ca.cert.pem

// The path to trusted certificates used by the Pulsar client to authenticate with Pulsar brokers
brokerClientTrustCertsFilePath: /path/to/ca.cert.pem
^^^ Can you share your values for these function worker configs?
d
not sure what to share
so this is the function log :
Copy code
WARN  org.apache.pulsar.client.impl.ClientCnx - [pulsar-hostanme/18.1.1.1:6651] Got exception io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeExc
eption: error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
and in the proxies i see
Copy code
Caused by: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
when i start the workers they first go to the broker on 8443
and this works
so i asume the function consumer does not provide the SSL cert
but this is configures in the functions_workers.yml
tried all day to work around this but with no success
d
TL;DR the function workers need to connect to the Pulsar brokers over TLS, so they need those configs I showed above all set to achieve that, eg. the broker url should be https+ssl://server:6651, etc
Since the proxies are saying that the peer (broker) didnā€™t return a cert, then most likely either you arenā€™t connecting to the TLS enabled endpoint of the broker, or the brokerā€™s TLS isnā€™t configured properly.
d
everything is working in the cluster
just added the standalone function wokers machines
that are not working
Copy code
############################################
# security settings for pulsar broker client
############################################
# The path to trusted certificates used by the Pulsar client to authenticate with Pulsar brokers
brokerClientTrustCertsFilePath: /opt/pulsar/ssl/ca.pem
# Whether to enable the broker client authentication used by function workers to talk to brokers
brokerClientAuthenticationEnabled: true
# the authentication plugin to be used by the pulsar client used in worker service
brokerClientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationTls
# the authentication parameter to be used by the pulsar client used in worker service
brokerClientAuthenticationParameters: tlsCertFile:/opt/pulsar/ssl/certificate.pem,tlsKeyFile:/opt/pulsar/ssl/key.pem
also the this
and :
Copy code
# Configure the pulsar client used by function metadata management
#
# points
# Whether to enable TLS when clients connect to broker
useTls: true
# For TLS:
# brokerServiceUrl=<pulsar+ssl://localhost:6651/>
pulsarServiceUrl: <pulsar+ssl://pulsar-hostname:6651>
# For TLS:
# webServiceUrl=<https://localhost:8443/>
pulsarWebServiceUrl: <https://pulsar-hostname:8443>
d
ok thanks. Those all look reasonable and correct
which version of Pulsar are you running?
d
so when connecting to the broekers so set the public/metadata on 8443 it works
but when the function consumer tries to connect on ort 6551 it says the above
proxy shows no cert in present in the request
2.10.2
d
what about connecting directly to port 6651 on the broker from a pulsar client, e.g. the 
pulsar-client
CLI? If you pass in the TLS certs, does the client connect or get the same error?
Stated differently, do your current clients have any issues communicating over the 6651 port to the broker?
d
yes, all clients can connect
šŸ¤” 1
d
Do you have the following values set in the function-worker config? I saw that you have the correct service URL, but you didnā€™t mention these.
Copy code
tlsEnabled: true
tlsCertificateFilePath: /path/to/functions-worker.cert.pem
tlsKeyFilePath:         /path/to/functions-worker.key-pk8.pem
tlsTrustCertsFilePath:  /path/to/ca.cert.pem

// The path to trusted certificates used by the Pulsar client to authenticate with Pulsar brokers
brokerClientTrustCertsFilePath: /path/to/ca.cert.pem
d
yep
pasted them above
Copy code
#### tls configuration for worker service
# Enable TLS
tlsEnabled: true
# Path for the TLS certificate file
tlsCertificateFilePath: /opt/pulsar/ssl/certificate.pem
# Path for the TLS private key file
tlsKeyFilePath: /opt/pulsar/ssl/key.pem
# Path for the trusted TLS certificate file
tlsTrustCertsFilePath: /opt/pulsar/ssl/ca.pem
# Accept untrusted TLS certificate from client
tlsAllowInsecureConnection: false
# Whether server hostname must match the common name of the certificate
tlsEnableHostnameVerification: false
# Tls cert refresh duration in seconds (set 0 to check on every new connection)
tlsCertRefreshCheckDurationSec: 300
# Whether client certificates are required for TLS. Connections are rejected if the client
# certificate isn't trusted.
tlsRequireTrustedClientCertOnConnect: false
i will try to trobleshoot more today
maybe i get around it šŸ™‚ it has been an adventure to enable function workers
šŸ™ 1
d
what about 
brokerClientTrustCertsFilePath
?
d
yep
Copy code
############################################
# security settings for pulsar broker client
############################################
# The path to trusted certificates used by the Pulsar client to authenticate with Pulsar brokers
brokerClientTrustCertsFilePath: /opt/pulsar/ssl/ca.pem
# Whether to enable the broker client authentication used by function workers to talk to brokers
brokerClientAuthenticationEnabled: true
# the authentication plugin to be used by the pulsar client used in worker service
brokerClientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationTls
# the authentication parameter to be used by the pulsar client used in worker service
brokerClientAuthenticationParameters: tlsCertFile:/opt/pulsar/ssl/certificate.pem,tlsKeyFile:/opt/pulsar/ssl/key.pem
on init there is also the problem with the ZK metadata
on init the worker writes in ZK that the ZK servers are 127.0.0.,1
that seems like a bug also
Copy code
1
BKDL
{"1":{"str":"localhost:2181"},"2":{"str":"/ledgers"},"3":{"tf":1},"4":{"tf":0},"5":{"str":"localhost:2181"},"6":{"str":"localhost:2181"},"7":{"str":"localhost:2181"}}
this entry
even if the ZK servers are set correctly in the config
in order to proceed i needed to update in Zk by hand
just FYI
šŸ‘ 1
so to add more info, when you create a function, it adds an entry to the ZK in /pulsar/functions
the problem is that the Zk is not set correctly in this entry
Copy code
1
BKDL
{"1":{"str":"localhost:2181"},"2":{"str":"/ledgers"},"3":{"tf":1},"4":{"tf":0},"5":{"str":"localhost:2181"},"6":{"str":"localhost:2181"},"7":{"str":"localhost:2181"}}
it should add the configured ZK servers when creating a new function
cc @David K @bala rao
2 Views
Open in Slack
PreviousNext
Powered by