Skip to content
This repository was archived by the owner on Dec 21, 2021. It is now read-only.
This repository was archived by the owner on Dec 21, 2021. It is now read-only.

Connection freeze after a while #249

Open

Description

@aseyfpour

Hi Dear Streamrs

I am trying to fetch a large amount of data from about 9 months ago. I developed an app that subscribes to a stream with resent option.
On the first try, I used a wide timeframe for the resent option (from: 9 months ago timestamp, to: now) but it freezes after a while and it doesn't receive any message or throw an exception.
So on the second try, I used a loop to subscribe on a narrow timeframe (an hour), after that It fetches all the messages in this timeframe I will unsubscribe this timeframe and will subscribe to the next timeframe. But in this scenario, it freezes too.
I tried these on different networks and different devices, the result is the same as each other. It is not happening at a specific time or after some amount of messages, it is completely random.
How can I help you to investigate this issue?

Activity

  1. fonty1 commented on May 5, 2021

    @fonty1
    Contributor

    Hi @aseyfpour - Thank you for reporting the issue!

    It's quite likely to be a Streamr Network issue as we've been experiencing problems with resends lately. We're rewriting this core resend logic and it's a large engineering effort that is more a less complete, but requiring some testing. Instead of using websockets, the client will use a classic REST request for resends so it should be far more stable.

    I would say this issue will resolve in 1-3 weeks and I will update you when the client update arrives.

  2. aseyfpour commented on May 5, 2021

    @aseyfpour
    ContributorAuthor

    Thank you @fonty1.

  3. fonty1 commented on May 5, 2021

    @fonty1
    Contributor

    @aseyfpour while I think it's the resend issue I spoke to in the last message, I forgot to mention you should double check you're running the latest version - as there was a memleak fixed in the latest,

    https://github.com/streamr-dev/streamr-client-javascript/releases/tag/v5.2.1

  4. aseyfpour commented on May 6, 2021

    @aseyfpour
    ContributorAuthor

    @fonty1 Yes, I'm using 5.2.1. I also tried the 5.3.0 alpha version. Nothing has changed.

    P.S: I reported that memleak issue: #233

  5. aseyfpour commented on May 6, 2021

    @aseyfpour
    ContributorAuthor

    I also mentioned these network issues when I was verifying @timoxley patch.

  6. aseyfpour commented on May 22, 2021

    @aseyfpour
    ContributorAuthor

    @fonty1 Is there any update on this topic?

  7. fonty1 commented on May 23, 2021

    @fonty1
    Contributor

    @fonty1 Is there any update on this topic?

    Hey @aseyfpour, we're finishing up internal testing of the resend upgrade. Fingers crossed for a release this week, thanks for your patience with this one!

  8. fonty1 commented on May 25, 2021

    @fonty1
    Contributor

    Hi @aseyfpour could you try your resends now? The resend data brokers have been updated. If you have issues, can you be so kind as to report the behaviour and any logs you get.

    Thanks!

  9. aseyfpour commented on May 25, 2021

    @aseyfpour
    ContributorAuthor

    Hi @fonty1, Sure. I will do that.
    Do I need to update my streamr-client? If yes, which version?

  10. fonty1 commented on May 25, 2021

    @fonty1
    Contributor

    @aseyfpour - please use v5.4.1 but the main change was on the backend, so it hopefully fixed in older versions as well.

  11. aseyfpour commented on May 25, 2021

    @aseyfpour
    ContributorAuthor

    @fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
    My Subscribe pseudo-code is something like this:

    client.subscribe({
    	streamId: stream_id,
    	resend: {
    		from: {
    			timestamp: x
    		},
    		to: {
    			timestamp: x + 1hr,
    		}
    	}
    }, async (message, metadata) => {
    	count++;
    	console.log(`Count: ${count}`)
    }).then(() => {
    	console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
    }).catch((err) => {
    	console.log(`Error occured ${stream_id}`)
    })
    

    When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).

  12. fonty1 commented on May 26, 2021

    @fonty1
    Contributor

    @fonty1 - First try result: It fetches about 17K messages in 45 minutes, then it gets stuck at Subscribe again. I didn't catch any exceptions or successful messages in my code. I think its Promise didn't resolve or was rejected by the client.
    My Subscribe pseudo-code is something like this:

    client.subscribe({
    	streamId: stream_id,
    	resend: {
    		from: {
    			timestamp: x
    		},
    		to: {
    			timestamp: x + 1hr,
    		}
    	}
    }, async (message, metadata) => {
    	count++;
    	console.log(`Count: ${count}`)
    }).then(() => {
    	console.log(`Subscribed to stream ${stream_id}, time window ${x} - ${x + 1hr}`)
    }).catch((err) => {
    	console.log(`Error occured ${stream_id}`)
    })
    

    When I restart my app, it started to fetch messages again from breaking point (faster than before, 120K messages in 25 minutes).

    @aseyfpour - Argh! Sorry to hear that. Can you share the streamId with me? Either here or to matthew.fontana@streamr.network. We can investigate the logs from there to see what went wrong.

  13. aseyfpour commented on May 26, 2021

    @aseyfpour
    ContributorAuthor

    @fonty1 - here is my stream id: 94ylzZvOTdqf_kHFkLT2Aw

  14. harbu commented on May 28, 2021

    @harbu
    Contributor

    Hey, you could try to directly request the data from this HTTP endpoint

    https://streamr.network/api/v1/streams/94ylzZvOTdqf_kHFkLT2Aw/data/partitions/0/range?fromTimestamp=<START_TIME>&toTimestamp=<END_TIME>
    

    replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.

    You also need to pass the following authentication header with the request:

    Authorization: Bearer <SESSION_TOKEN>
    

    The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.

  15. hpihkala commented on May 28, 2021

    @hpihkala
    Contributor

    While the workaround above should work for the time being, we're also investigating the original issue why the resends over websocket get corrupted. Could be an issue with the websocket library we're using.

  16. aseyfpour commented on Jun 1, 2021

    @aseyfpour
    ContributorAuthor

    Hey, you could try to directly request the data from this HTTP endpoint

    https://streamr.network/api/v1/streams/94ylzZvOTdqf_kHFkLT2Aw/data/partitions/0/range?fromTimestamp=<START_TIME>&toTimestamp=<END_TIME>
    

    replacing <START_TIME> and <END_TIME> with Unix milliseconds since EPOCH times.

    You also need to pass the following authentication header with the request:

    Authorization: Bearer <SESSION_TOKEN>
    

    The <SESSION_TOKEN> for your wallet can be obtained with streamr-client-javascript by calling streamrClient.getSessionToken(). Note that the session token expires after some time.

    Sure, I will feedback to you soon.

  17. aseyfpour commented on Jun 16, 2021

    @aseyfpour
    ContributorAuthor

    @harbu The Rest API is OK. Its connection slows down sometimes but it is much reliable than the Streamr client.

  18. aseyfpour commented on Jul 4, 2021

    @aseyfpour
    ContributorAuthor

    Any update on this?

  19. fonty1 commented on Jul 5, 2021

    @fonty1
    Contributor

    @aseyfpour - It's been a bit of a rabbit hole as multiple dependencies have needed to be switched out, I'll see about getting an update from the Network team this week about how much work is left before we can make a new production release.

  20. fonty1 commented on Jul 15, 2021

    @fonty1
    Contributor

    @aseyfpour - could you try again now? We've made some very heavy updates to the broker that are very fresh, but may solve the issue. Please give it a go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions