Skip to content

Some doubt towards the fix_mask setting in diffusion module #19

Closed

Description

@Plankson

Thanks for your solid work on diffusion for decision-making.
Here I have a small question about the fix_mask setting. How does the diffusion model perform if I change the fix_mask during inference? For instance, in previous baselines such as diffuser, the fix_mask is consistently set to the state at horizon=0 throughout the entire experiment. However, if I want to generate the state-action sequence given the initial state s_0 and the final state s_{#horizon-1}, can I easily modify the fix_mask during the inference stage to get desired performance? Or should I retrain the diffusion model with the new fix_mask?

Activity

  1. changed the title [-]Some doubt problem towards the fix_mask setting in diffusion module[/-] [+]Some doubt towards the fix_mask setting in diffusion module[/+] on Jul 11, 2024
  2. ZibinDong commented on Jul 11, 2024

    @ZibinDong
    Contributor

    Thank you for using CleanDiffuser and providing feedback. : )
    Since fix_mask affects model training, we cannot change the fix_mask during inference. Therefore, we do need to retrain the diffusion model with the new fix_mask.

    Simply try the fix_mask below and train your new diffusion model. : )

    fix_mask = torch.zeros((horizon, obs_dim + act_dim))
    fix_mask[[0, -1], :obs_dim] = 1.
  3. Plankson commented on Jul 11, 2024

    @Plankson
    Author

    I see. Thanks for your reply! :)

  4. pinned this issue on Jul 11, 2024
  5. nkalyanv99 commented on Jan 2, 2025

    @nkalyanv99

    Hi, how would you perform closed loop planning at test time though for this in-painting case? Once you generate a trajectory, take a step. The new fixed_mask should now be fix_mask[0, (-2)] = 1. (Since now you've H-1 steps left while rolling out?)
    And this would change every step?

  6. ZibinDong commented on Jan 2, 2025

    @ZibinDong
    Contributor

    Hi, how would you perform closed loop planning at test time though for this in-painting case? Once you generate a trajectory, take a step. The new fixed_mask should now be fix_mask[0, (-2)] = 1. (Since now you've H-1 steps left while rolling out?) And this would change every step?

    Hello, let me explain.

    I believe the "final state" mentioned by Plankson specifically refers to s_{horizon-1}, rather than the "goal" in a goal-conditioned RL setting. Therefore, if we have a way to continuously indicate the state that needs to be reached after H steps, then the fix_mask would constantly equal 1 on [0, -1].

    If you want to implement a planning method that allows the agent to move toward a given "goal" state, you might try modifying the training data to make the last state of trajectories to be generated become the goal, i.e., s_{-1} = goal_state. This way, you can do closed-loop planning using inpainting during inference.

    As for the approach you proposed, I believe it's certainly possible, but it requires us to know how many steps are needed to reach the goal state, which isn't very practical.

  7. nkalyanv99 commented on Jan 2, 2025

    @nkalyanv99

    Thank you very much for the quick reply! Yes indeed at train time we can set the last element as the goal and use a fix_mask of [0,-1].

    I didn't really follow how we'd do inpainting at inference time though, at timestep t=0 the fix_mask makes sense. But after taking a step what would our prior and fix_mask be to generate the next action?

  8. ZibinDong commented on Jan 3, 2025

    @ZibinDong
    Contributor

    Thank you very much for the quick reply! Yes indeed at train time we can set the last element as the goal and use a fix_mask of [0,-1].

    I didn't really follow how we'd do inpainting at inference time though, at timestep t=0 the fix_mask makes sense. But after taking a step what would our prior and fix_mask be to generate the next action?

    I think each step is the same, like,

    at timestep t=0, we observe state $s_0$ from env and get the goal $s_{goal}$ from a certain model, then we set

    prior[0], prior[-1] = s_0, s_goal
    fix_mask[[0, -1], :obs_dim] = 1.

    we sample an action to execute

    traj, _ = diffusion.sample(...)
    a_0 = traj[0, 0, obs_dim:]. # suppose that batch_size=1
    s_1, r, done, info = env.step(a_0.numpy().cpu())

    at timestep t=1, we observe state $s_1$ from env and get the goal $s_{goal}$ from a certain model, then we set

    prior[0], prior[-1] = s_1, s_goal
    fix_mask[[0, -1], :obs_dim] = 1.

    we do the same sampling step to get action and execute.

    So fix_mask never changes and we only reset the first element in prior to tell the model the current observation.

  9. nkalyanv99 commented on Jan 3, 2025

    @nkalyanv99

    Thanks again for the detailed reply! But according to this won't the horizon to reaching goal always be H? (That is even after taking a step in the environment, you're still telling the model you've to reach the goal after H more steps)

  10. ZibinDong commented on Jan 3, 2025

    @ZibinDong
    Contributor

    Thanks again for the detailed reply! But according to this won't the horizon to reaching goal always be H? (That is even after taking a step in the environment, you're still telling the model you've to reach the goal after H more steps)

    Yeah, that's true. So if you want to perform planning in this inpainting style, you have to suppose you have a upper model that can tell which state to reach after H steps, like, https://arxiv.org/abs/2401.15443. Otherwise, I think this setting is indeed unreasonable.

  11. nkalyanv99 commented on Jan 3, 2025

    @nkalyanv99

    I see that makes sense. Thanks for clarifying!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions