[Feature]: Integrate ComfyUI for local workflow execution and routing #182

Area

Engine or model management

User problem

at the moment, using ComfyUI across multiple machines means choosing which instance to use, checking that it has the right models and custom nodes, moving inputs over and collecting the outputs afterwards. PAIR already handles discovery, pairing and inference routing, so it feels like a useful place to bring this together.

This could help people whose own computer has limited GPU compute but who have access to another machine or shared school/lab hardware. They could build workflows on their computer and send generation jobs to available workers. Each workflow would still need to fit on the worker running it.

Desired outcome

• Connect an existing ComfyUI installation and see whether it’s ready.
• Open the ComfyUI editor through a local PAIR endpoint.
• Submit workflows to a paired machine that can run them.
• See queue status and progress, cancel jobs and get their outputs back.
• Get a useful explanation when a machine can’t run something, such as missing models, custom nodes or insufficient memory.

Each workflow would run entirely on one worker, with independent workflows able to run on different workers at the same time.

For a starting point, i was thinking of a small Go adapter that submits an API-format workflow to one existing ComfyUI instance and maps its execution status. Engine management, editor support, file transfers and automatic routing could then come in seperate PRs.

Alternatives considered

• Use each ComfyUI instance directly. This works, but users still have to choose the machine and handle files and compatibility themselves.
• Add a ComfyUI node that calls PAIR’s LLM endpoints. Useful for things like prompt expansion, but it wouldn’t distribute the actual image/video generation.
• Use a generic HTTP proxy. This would handle forwarding, but wouldn’t know whether a worker can run a workflow or keep track of it after submission.

Compatibility and security implications

A few things I want to account for:

• Keep existing Ollama, LM Studio and llama.cpp behaviour working.
• Use PAIR’s existing local endpoints and authenticated communication between peers.
• Keep uploads, progress, cancellation and outputs tied to the worker running that job.
• Treat an accepted submission separately from a completed generation.
• Check uncertain submissions before retrying, so the same workflow doesn’t accidentally run twice.
• Define which ComfyUI versions and custom-node setups are supported.
• Keep prompts, workflow contents and generated content out of operational logs.

For a start, we can target an existing, provisioned ComfyUI installation. Installation and update support could be agreed separately.

Validation approach

1. Test workflow submission and status mapping with a fake ComfyUI HTTP server.
2. Run a real workflow against one worker through PAIR and retrieve the output.
3. Check that editor uploads, progress events, previews and downloads work through the proxy.
4. Run independent workflows on two compatible workers and confirm they execute concurrently.
5. Check that incompatible workers are excluded with a useful explanation.
6. Test cancellation, connection failures, lost submission responses, restarts and interrupted downloads.
7. Run the existing routing tests and service-contract checks.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

@Noah-Tervalon-Nvidia, would the team be open to a community contribution towards ComfyUI support? I saw it in the roadmap. I’m working on something similar for a school project and would be interested in contributing the work if possible. Before getting too far into it, I’d appreciate your thoughts on this! Thank you.