Skip to content
This repository was archived by the owner on Jul 30, 2026. It is now read-only.
This repository was archived by the owner on Jul 30, 2026. It is now read-only.

Application: Data Processing Workflows on IPFS #58

Open

Description

@flyingzumwalt

Work in progress - please contribute. See #40.

Essential Use Case: when running data processing/analysis workflows, use IPFS as the storage layer. This allows your workflows to be agnostic about where the data are stored -- pulling all the source data onto a local node before running a workflow is an optimisation choice that can be done on the fly with zero impact on the code. Likewise, the results of the workflows can be written to IPFS and moved around as needed without impacting the referential integrity of your data.

Activity

  1. jbenet commented on Apr 5, 2017

    @jbenet
    Member

    Data Processing Workflows on IPFS

    Essential Use Case

    • use ipfs as a source or sink in data processing pipelines

    Use Cases

    Concrete

    • Alice wants to render all tiles in a world map, using hundreds of worker computers.
    • Bob wants to run calculations over a stream of geo-tagged events, generated by millions of users.
    • Charlie wants to make available a down-sampled, filtered, and versioned archive of astrophysics photography.
    • Dana wants to train a deep-learning object classifier for use in self-driving cars.
    • Eve wants to run speech-to-text transcription and NLP tagging over large volumes of audio data.
    • Faye wants to run tests (and collect results) for her distributed file system on thousands of machines around the world.

    Groupings:

    • MapReduce / Hadoop use cases (IPFS takes the role of GFS or HDFS)
    • Spark use cases (IPFS takes the role of data source & sink)
    • Real-time data stream processing (ingest data into IPFS, process it as it appears)
    • Versioning of datasets processed
    • Storing and distributing datasets processed
    • P2P backbone for a distributed computing cluster
    • Scientists and Data Scientists running data experiments
      • in part. Machine Learning, Bioinformatics, Astronomy, etc.

    Other:

    • IPFS can be source and sink for data
    • IPFS can store intermediate results
    • IPFS can deduplicate data and maybe work
    • IPFS can version all results
    • libp2p pubsub can be used to announce events

    Foundational Features + Functionality

    • ipfs basics (add, cat)
    • ipfs data importers (for better perf and dedup)
    • ipfs versioning
    • very high throughput & perf
    • libp2p pubsub (to announce events)
    • libp2p pluggable routing to have fast content-routing

    Existing Projects + Organizations Working in this Area

    There have been a number of people who have expressed desire to:

    1. Our own test lab: https://github.com/ipfs/test-lab
    2. things like BrowserCloud, Pando, etc.
    3. things like Golem Project
  2. removed their assignment
    on Apr 5, 2017
  3. 0zAND1z commented on May 26, 2018

    @0zAND1z

    Read this very interesting architecture: https://www.cse.unsw.edu.au/~hpaik/thesis/showcases/16s2/scott_brisbane.pdf

    Planning on building the full scale services..

  4. saurabhdhupar commented on May 27, 2018

    @saurabhdhupar
  5. 0zAND1z commented on May 28, 2018

    @0zAND1z

    Thanks for adding the link to the full report

  6. akevy commented on Jun 4, 2018

    @akevy

    Great, we're trying

  7. echarles commented on Jun 17, 2018

    @echarles

    @scottybrisbane do you have a public repo with your ipfs/hdfs integration?

  8. scottybrisbane commented on Jun 18, 2018

    @scottybrisbane

    @echarles not just yet, although I am planning to post my work. It's very much a POC, but could be a good starting point for anyone wanting to get something going.

    I'll update this thread when I post the code.

  9. echarles commented on Jun 18, 2018

    @echarles

    Thx @scottybrisbane - POC is very fine. Once published, I expect contributors (like me) to try and let evolve the code. Without pushing you, any ideas on the timeline? (do we speak about days, weeks, months... before having something public?). Btw If you fear uncompleted feature, not-perfect code, no documentation... just push what you have and other will help, that's how opensource works.

  10. scottybrisbane commented on Jun 18, 2018

    @scottybrisbane

    @echarles I'm hoping to have it up within a few weeks.

  11. bo-liu commented on Sep 30, 2018

    @bo-liu

    Hey Folks, Any update on this topic? @scottybrisbane @echarles

  12. ajbouh commented on Sep 30, 2018

    @ajbouh
  13. bo-liu commented on Oct 9, 2018

    @bo-liu

    Hi, can I ask what sort of data processing people want to use IPFS for?

    Really depends on what kind of data(or which kind of data you are interested in) stored inside IPFS.

  14. ajbouh commented on Oct 9, 2018

    @ajbouh

    Right, I'm looking to help make sure IPFS is a good fit for the kind of data processing people want to do.

    Getting specific examples helps me ensure we're putting effort in the right places.

  15. bertrandfalguiere commented on Nov 16, 2018

    @bertrandfalguiere

    @scottybrisbane Your work is really interesting! The use case is fascinating.
    Do you know when you will be able to post your work?

  16. 0zAND1z commented on Nov 17, 2018

    @0zAND1z

    KIP team(@KIPFoundation) is working on one of the reference implementation that may align with Scotty's work.
    More about our implementation of big data persistence will be updated under section 7 here: https://kipfoundation.github.io/techprimer/7-Realm-Storage.html

    Look forward to adding HDFS support together!

  17. Duske commented on Apr 5, 2019

    @Duske

    @scottybrisbane Great work! Are you going to publish your code? I would really like to make use of it in my own thesis, so if you need some help, just hit me up :)

  18. jessicaschilling commented on Mar 26, 2020

    @jessicaschilling

    Note: Discussion on applications of IPFS are happening over in the IPFS Forums now ... please continue the discussion there!

    This issue is being moved over to the archived repo https://github.com/ipfs/apps/ for reference.

  19. transferred this issue fromipfs/ipfson Mar 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions