This repository was archived by the owner on Apr 12, 2019. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 12, 2019. It is now read-only.
Data Maintance #79
Description
Hey guys,
one thing that might still be open: can the scraping cli command be run multiple times without inserting doubles?
We want to maintain the data in the coming 2-3 years at least. So that running the CLI multiple times should ensure that it checks if the rows are already in the database and update the entity accordingly.
We want to put this into a cronjob and run each week or month.
Thanks
Reacted by Christian Dallago
Activity
sacdallago commentedon Apr 18, 2017
http://docs.sequelizejs.com/en/latest/api/model/#upsertvalues-options-promisecreated might be a place to start, substituting inserts, but only if you have a logic in place to match two documents. You might eventually need to create a distance between to objects dictated by the object's fields (or features) and define a threshold by which you consider two objects similar :) but, but. For now, I would say that
upsertinstead of create should do, based on name & birthdate matching?