Skip to content

Update client-server RL, distillation, and checkpoint workflows - #403

Draft
Yunnglin wants to merge 22 commits into
dev-swift-v5from
fix/transformers-task-head-checkpoint
Draft

Yunnglin wants to merge 22 commits into
dev-swift-v5from
fix/transformers-task-head-checkpoint

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

PR type

  • Bug Fix
  • New Feature
  • Document Updates

PR information

  • Update the client GRPO recipes for single-turn GSM8K and multi-turn Blackjack. Use StreamingDriver with standard vLLM sampling, and drain rollout calls before releasing training handles. The GSM8K example no longer needs a DataPlane deployment.
  • Add SDK examples for online GKD, OPD-RL and OPSD. Support teacher logits and continuation scoring over HTTP, and configure AdamW in Megatron's optimizer.
  • Preserve task-head and LoRA checkpoint snapshots, fix Ray worker startup and simplify Serve routing. Report ModelScope upload progress and revisions, and let clients reconnect to the same upload task.

Experiment results

  • 100 focused offline tests passed on this branch, including GRPO, resource release, continuation scoring, and upload handling. The example server config and both documentation builds passed.
  • The preliminary PPU4 run returned 26 real GSM8K samples without DataPlane. The sampler stopped during generation, before any optimizer update. Real optimizer updates and sampling with the newly published LoRA still need verification.

Supersedes #402 and includes the existing changes on the original feature branch.

Keep mock template setup independent of model assets so the sampler smoke tests can run offline.
Migrate the three Blackjack entry points to the session rollout API and document CPU client setup and training output.
Require one Processor replica and close expired Dataset and DataLoader workers. Preserve authentication, session IDs, request IDs, and worker plugin setup.

Remove redundant training session lookups and exercise Processor cleanup and both SDKs over HTTP.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant