Repository navigation
Reader set to MessageId.latest and inclusive start message does not work #193
Description
Hi team,
When using start_message_id_inclusive, we expect the message seeked to be returned. This is the case for the pulsar.MessageId.earliest, but does not work with pulsar.MessageId.latest.
Reproduction
First, run a pulsar standalone instance:
docker run -it -p 6650:6650 -p 8080:8080 --tmpfs /pulsar/data apachepulsar/pulsar:3.1.0 bin/pulsar standaloneThen, run this code:
import pulsar
host = "pulsar://localhost:6650"
topic = "example"
client = pulsar.Client(host)
producer = client.create_producer(topic)
producer.send(b"Hello world 1!")
producer.send(b"Hello world 2!")
earliest_reader = client.create_reader(topic, pulsar.MessageId.earliest, start_message_id_inclusive=True)
msg = earliest_reader.read_next()
print(msg.value())
# Prints Hello world 1!, as wanted
latest_reader = client.create_reader(topic, pulsar.MessageId.latest, start_message_id_inclusive=True)
msg = latest_reader.read_next(timeout_millis=5000)
print(msg.value())
# Times outThe behaviour of the earliest reader is as expected. But the latest reader should not time out.
Metadata
Metadata
Assignees
Labels
No labels
Activity
RobertIndie commentedon Jan 11, 2024
The
latestmeans that you are reading the messages that are produced after the reader is created. It shouldn't work withstart_message_id_inclusive.Samreay commentedon Jan 11, 2024
That seems incredibly unintuitive. If I say to someone "Hey, get me the latest message" I'd expect them to give me the latest message, not the first message produced after I asked.
Be that as it may, how should I go about consuming the last message in a topic with the python client then? Right now I'm seeking to a day in the past and just reading everything until the final message, which is incredibly wasteful.
treuherz commentedon Jan 11, 2024
@RobertIndie That doesn't seem right. The Go client explicitly handles the case where
startMessageIDInclusive && startingMessageID == latestMessageID. I would expect the Python client to match that behaviour or at least document that the functionality isn't working yet.Samreay commentedon Jan 11, 2024
Yeah looks like the Java client also checks to see if its latest and inclusive, and then seeks explicitly to the last message: https://github.com/apache/pulsar/blob/176bdeacd309e8c1e49358987a1946abd30ba34a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L2362
RobertIndie commentedon Jan 11, 2024
Thanks for all your information.
That makes sense to me.
RobertIndie commentedon Jan 12, 2024
This issue is related to this C++ client issue: apache/pulsar-client-cpp#385. I have pushed a PR to fix it: apache/pulsar-client-cpp#386.
Hopefully it will be released in the next feature release of the Python client.
Add tests for apache#193
Upgrade the C++ client to 3.5.0 for some bug fixes