You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Dec 21, 2021. It is now read-only.
Repository navigation
This repository was archived by the owner on Dec 21, 2021. It is now read-only.
I am trying to fetch a large amount of data from about 9 months ago. I developed an app that subscribes to a stream with resent option.
On the first try, I used a wide timeframe for the resent option (from: 9 months ago timestamp, to: now) but it freezes after a while and it doesn't receive any message or throw an exception.
So on the second try, I used a loop to subscribe on a narrow timeframe (an hour), after that It fetches all the messages in this timeframe I will unsubscribe this timeframe and will subscribe to the next timeframe. But in this scenario, it freezes too.
I tried these on different networks and different devices, the result is the same as each other. It is not happening at a specific time or after some amount of messages, it is completely random.
How can I help you to investigate this issue?
Hi @aseyfpour - Thank you for reporting the issue!
It's quite likely to be a Streamr Network issue as we've been experiencing problems with resends lately. We're rewriting this core resend logic and it's a large engineering effort that is more a less complete, but requiring some testing. Instead of using websockets, the client will use a classic REST request for resends so it should be far more stable.
I would say this issue will resolve in 1-3 weeks and I will update you when the client update arrives.
@aseyfpour while I think it's the resend issue I spoke to in the last message, I forgot to mention you should double check you're running the latest version - as there was a memleak fixed in the latest,
Hey @aseyfpour, we're finishing up internal testing of the resend upgrade. Fingers crossed for a release this week, thanks for your patience with this one!
Hi @aseyfpour could you try your resends now? The resend data brokers have been updated. If you have issues, can you be so kind as to report the behaviour and any logs you get.
@fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
My Subscribe pseudo-code is something like this:
client.subscribe({
streamId: stream_id,
resend: {
from: {
timestamp: x
},
to: {
timestamp: x + 1hr,
}
}
}, async (message, metadata) => {
count++;
console.log(`Count: ${count}`)
}).then(() => {
console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
}).catch((err) => {
console.log(`Error occured ${stream_id}`)
})
When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).
@fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
My Subscribe pseudo-code is something like this:
client.subscribe({
streamId: stream_id,
resend: {
from: {
timestamp: x
},
to: {
timestamp: x + 1hr,
}
}
}, async (message, metadata) => {
count++;
console.log(`Count: ${count}`)
}).then(() => {
console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
}).catch((err) => {
console.log(`Error occured ${stream_id}`)
})
When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).
@aseyfpour - Argh! Sorry to hear that. Can you share the streamId with me? Either here or to matthew.fontana@streamr.network. We can investigate the logs from there to see what went wrong.
replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.
You also need to pass the following authentication header with the request:
Authorization: Bearer <SESSION_TOKEN>
The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.
While the workaround above should work for the time being, we're also investigating the original issue why the resends over websocket get corrupted. Could be an issue with the websocket library we're using.
replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.
You also need to pass the following authentication header with the request:
Authorization: Bearer <SESSION_TOKEN>
The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.
@aseyfpour - It's been a bit of a rabbit hole as multiple dependencies have needed to be switched out, I'll see about getting an update from the Network team this week about how much work is left before we can make a new production release.
@aseyfpour - could you try again now? We've made some very heavy updates to the broker that are very fresh, but may solve the issue. Please give it a go.
Activity
fonty1 commentedon May 5, 2021
Hi @aseyfpour - Thank you for reporting the issue!
It's quite likely to be a Streamr Network issue as we've been experiencing problems with resends lately. We're rewriting this core resend logic and it's a large engineering effort that is more a less complete, but requiring some testing. Instead of using websockets, the client will use a classic REST request for resends so it should be far more stable.
I would say this issue will resolve in 1-3 weeks and I will update you when the client update arrives.
aseyfpour commentedon May 5, 2021
Thank you @fonty1.
fonty1 commentedon May 5, 2021
@aseyfpour while I think it's the resend issue I spoke to in the last message, I forgot to mention you should double check you're running the latest version - as there was a memleak fixed in the latest,
https://github.com/streamr-dev/streamr-client-javascript/releases/tag/v5.2.1
aseyfpour commentedon May 6, 2021
@fonty1 Yes, I'm using 5.2.1. I also tried the 5.3.0 alpha version. Nothing has changed.
P.S: I reported that memleak issue: #233
aseyfpour commentedon May 6, 2021
I also mentioned these network issues when I was verifying @timoxley patch.
aseyfpour commentedon May 22, 2021
@fonty1 Is there any update on this topic?
fonty1 commentedon May 23, 2021
Hey @aseyfpour, we're finishing up internal testing of the resend upgrade. Fingers crossed for a release this week, thanks for your patience with this one!
fonty1 commentedon May 25, 2021
Hi @aseyfpour could you try your resends now? The resend data brokers have been updated. If you have issues, can you be so kind as to report the behaviour and any logs you get.
Thanks!
aseyfpour commentedon May 25, 2021
Hi @fonty1, Sure. I will do that.
Do I need to update my streamr-client? If yes, which version?
fonty1 commentedon May 25, 2021
@aseyfpour - please use v5.4.1 but the main change was on the backend, so it hopefully fixed in older versions as well.
aseyfpour commentedon May 25, 2021
@fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
My Subscribe pseudo-code is something like this:
When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).
fonty1 commentedon May 26, 2021
@aseyfpour - Argh! Sorry to hear that. Can you share the streamId with me? Either here or to matthew.fontana@streamr.network. We can investigate the logs from there to see what went wrong.
aseyfpour commentedon May 26, 2021
@fonty1 - here is my stream id: 94ylzZvOTdqf_kHFkLT2Aw
harbu commentedon May 28, 2021
Hey, you could try to directly request the data from this HTTP endpoint
replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.
You also need to pass the following authentication header with the request:
The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling
streamrClient.getSessionToken(). Note that the session token expires after some time.hpihkala commentedon May 28, 2021
While the workaround above should work for the time being, we're also investigating the original issue why the resends over websocket get corrupted. Could be an issue with the websocket library we're using.
aseyfpour commentedon Jun 1, 2021
Sure, I will feedback to you soon.
aseyfpour commentedon Jun 16, 2021
@harbu The Rest API is OK. Its connection slows down sometimes but it is much reliable than the Streamr client.
aseyfpour commentedon Jul 4, 2021
Any update on this?
fonty1 commentedon Jul 5, 2021
@aseyfpour - It's been a bit of a rabbit hole as multiple dependencies have needed to be switched out, I'll see about getting an update from the Network team this week about how much work is left before we can make a new production release.
fonty1 commentedon Jul 15, 2021
@aseyfpour - could you try again now? We've made some very heavy updates to the broker that are very fresh, but may solve the issue. Please give it a go.