High memory usage while syncing #1576

Activity

1. seonpyoKim commented on May 29, 2019

   

   seonpyoKim

   on May 29, 2019

   ContributorAuthor

   More actions

   Now I'm suspecting this comes from "not shrinking" the Downloaded like data implemented in HeaderDownloader and BodyDownloader. I checked that length of queued (see #1567, data of which was previously included in downloaded) reaches up to about 900,000.
   That had not shrunk and will be alive continuously, I think the HashMap occupied huge amount of memory.
2. seonpyoKim commented on May 29, 2019

   

   seonpyoKim

   on May 29, 2019

   ContributorAuthor

   More actions

   HashMap has shrink_to_fit method, But I'm worrying about the reallocation overhead.
   @foriequal0 showed me that the reallocation step occurs sparsely with doubling the previous size when inserting to HashMap but occurs densely when shrinking the HashMap.
3. sgkim126

   added

   sync

   on May 30, 2019
4. sgkim126 commented on May 30, 2019

   

   sgkim126

   on May 30, 2019

   Contributor

   More actions

   Hi @HoOngEe,
   Please call shirink_to_fit after removing items.
   I don't think it becomes a burden to reallocate because HashMap allocates memory more than it needs when there is no capacity.
   And please apply this change to HashSet and Vec in BodyDownloader.
5. seonpyoKim commented on Aug 28, 2019

   

   seonpyoKim

   on Aug 28, 2019

   ContributorAuthor

   More actions

   Here is the main reason of high memory. #1585 (comment)
6. majecty

   closed this as completedin #1755on Aug 30, 2019