You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 30, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Jul 30, 2026. It is now read-only.
Application: Data Processing Workflows on IPFS #58
Essential Use Case: when running data processing/analysis workflows, use IPFS as the storage layer. This allows your workflows to be agnostic about where the data are stored -- pulling all the source data onto a local node before running a workflow is an optimisation choice that can be done on the fly with zero impact on the code. Likewise, the results of the workflows can be written to IPFS and moved around as needed without impacting the referential integrity of your data.
Reacted by leonardo A. janz and Alfonso Montero López
Reacted by Whyrusleeping, Nirmal Thacker, Wildan Zulfikar, Guilherme ( coder ) de Souza and Alfonso Montero López
@echarles not just yet, although I am planning to post my work. It's very much a POC, but could be a good starting point for anyone wanting to get something going.
Thx @scottybrisbane - POC is very fine. Once published, I expect contributors (like me) to try and let evolve the code. Without pushing you, any ideas on the timeline? (do we speak about days, weeks, months... before having something public?). Btw If you fear uncompleted feature, not-perfect code, no documentation... just push what you have and other will help, that's how opensource works.
@scottybrisbane Great work! Are you going to publish your code? I would really like to make use of it in my own thesis, so if you need some help, just hit me up :)
Activity
jbenet commentedon Apr 5, 2017
Data Processing Workflows on IPFS
Essential Use Case
Use Cases
Concrete
Groupings:
Other:
Foundational Features + Functionality
Existing Projects + Organizations Working in this Area
There have been a number of people who have expressed desire to:
0zAND1z commentedon May 26, 2018
Read this very interesting architecture: https://www.cse.unsw.edu.au/~hpaik/thesis/showcases/16s2/scott_brisbane.pdf
Planning on building the full scale services..
saurabhdhupar commentedon May 27, 2018
@kggp1995
https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesis/decentralising-big-data-processing.pdf
0zAND1z commentedon May 28, 2018
Thanks for adding the link to the full report
akevy commentedon Jun 4, 2018
Great, we're trying
echarles commentedon Jun 17, 2018
@scottybrisbane do you have a public repo with your ipfs/hdfs integration?
scottybrisbane commentedon Jun 18, 2018
@echarles not just yet, although I am planning to post my work. It's very much a POC, but could be a good starting point for anyone wanting to get something going.
I'll update this thread when I post the code.
echarles commentedon Jun 18, 2018
Thx @scottybrisbane - POC is very fine. Once published, I expect contributors (like me) to try and let evolve the code. Without pushing you, any ideas on the timeline? (do we speak about days, weeks, months... before having something public?). Btw If you fear uncompleted feature, not-perfect code, no documentation... just push what you have and other will help, that's how opensource works.
scottybrisbane commentedon Jun 18, 2018
@echarles I'm hoping to have it up within a few weeks.
bo-liu commentedon Sep 30, 2018
Hey Folks, Any update on this topic? @scottybrisbane @echarles
ajbouh commentedon Sep 30, 2018
bo-liu commentedon Oct 9, 2018
Really depends on what kind of data(or which kind of data you are interested in) stored inside IPFS.
ajbouh commentedon Oct 9, 2018
Right, I'm looking to help make sure IPFS is a good fit for the kind of data processing people want to do.
Getting specific examples helps me ensure we're putting effort in the right places.
bertrandfalguiere commentedon Nov 16, 2018
@scottybrisbane Your work is really interesting! The use case is fascinating.
Do you know when you will be able to post your work?
0zAND1z commentedon Nov 17, 2018
KIP team(@KIPFoundation) is working on one of the reference implementation that may align with Scotty's work.
More about our implementation of big data persistence will be updated under section 7 here: https://kipfoundation.github.io/techprimer/7-Realm-Storage.html
Look forward to adding HDFS support together!
Duske commentedon Apr 5, 2019
@scottybrisbane Great work! Are you going to publish your code? I would really like to make use of it in my own thesis, so if you need some help, just hit me up :)
jessicaschilling commentedon Mar 26, 2020
Note: Discussion on applications of IPFS are happening over in the IPFS Forums now ... please continue the discussion there!
This issue is being moved over to the archived repo https://github.com/ipfs/apps/ for reference.
lookfirst commentedon Jan 14, 2023
https://github.com/filecoin-project/bacalhau/wiki