Repository navigation
Pulsar client with custom logger may not terminate gracefully when a producer is created #256
Description
When providing a custom logger to the Pulsar client and creating a producer, intermittent terminations occur with the following error:
terminate called without an active exception
Aborted (core dumped)This doesn't happen consistently; I'd estimate it occurs in about 1 out of 20 runs. The other executions complete gracefully as expected. As the setup here is almost exactly as in #194, I've tried adding the del statements suggested there, but that hasn't resolved the issue for me.
Here is the Python script I'm using:
import logging
import pulsar
def open_and_close():
pulsar_logger = logging.getLogger("pulsarDummyLogger")
client = pulsar.Client("pulsar://localhost:6650", logger=pulsar_logger)
producer = client.create_producer("A")
producer.close()
client.close()
del producer
del client
if __name__ == "__main__":
open_and_close()I'm running it using the following bash command (as it doesn't happen all the time, I must run it a few times to encounter it):
for i in {1..20}; do env/bin/python python_script.py || break; doneI tried it on python 3.11.3 and on 3.13.3, as well as with pulsar-client 3.5.0, 3.6.0, 3.6.1, 3.7.0.
Any insights or suggestions would be greatly appreciated
Metadata
Metadata
Assignees
Labels
No labels
Activity
BewareMyPower commentedon May 22, 2025
I cannot reproduce it on macOS or Ubuntu 22.04 docker container (
localhostis changed tohost.docker.internal).BewareMyPower commentedon May 22, 2025
Oh I can reproduced it with another round of tests with the INFO log enabled:
Let me assign it to myself
BewareMyPower commentedon May 22, 2025
It seems that there is a rare chance that the shutdown thread failed to exit.
From the logs above from GDB, LWP 6877, which is the shutdown thread introduced in apache/pulsar#15316, didn't exit when the exception was thrown.
std::thread shutdownTask{[this, self, callback] { shutdown(); // ...I added a log after
client.close()and enabled debug level logs:The most possible reason is that the shutdown thread didn't exit when the Python interpreter exited. It might need a fix from the C++ client side.
While I might need more time to figure out a fix and verify if it works, I thought of a more tricky workaround and found it worked.
Could you verify it?
tzvi-shir-vaknin commentedon May 22, 2025
Thanks for looking into this and suggesting a temporary solution! Your time and effort are greatly appreciated.
Yes, adding the timeout does resolve the immediate issue. Thank you!
For my specific use case, I'd prefer to stick with the default Pulsar logger at the ERROR level. This would keep logs minimal, appearing only in rare error scenarios, rather than implementing a timeout each time I close the client.
I hope you'll be able to address the root cause of this issue soon.
Thanks again!