You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Some doubt towards the fix_mask setting in diffusion module #19
Thanks for your solid work on diffusion for decision-making.
Here I have a small question about the fix_mask setting. How does the diffusion model perform if I change the fix_mask during inference? For instance, in previous baselines such as diffuser, the fix_mask is consistently set to the state at horizon=0 throughout the entire experiment. However, if I want to generate the state-action sequence given the initial state s_0 and the final state s_{#horizon-1}, can I easily modify the fix_mask during the inference stage to get desired performance? Or should I retrain the diffusion model with the new fix_mask?
changed the title [-]Some doubt problem towards the fix_mask setting in diffusion module[/-][+]Some doubt towards the fix_mask setting in diffusion module[/+]on Jul 11, 2024
Thank you for using CleanDiffuser and providing feedback. : )
Since fix_mask affects model training, we cannot change the fix_mask during inference. Therefore, we do need to retrain the diffusion model with the new fix_mask.
Simply try the fix_mask below and train your new diffusion model. : )
Hi, how would you perform closed loop planning at test time though for this in-painting case? Once you generate a trajectory, take a step. The new fixed_mask should now be fix_mask[0, (-2)] = 1. (Since now you've H-1 steps left while rolling out?)
And this would change every step?
Hi, how would you perform closed loop planning at test time though for this in-painting case? Once you generate a trajectory, take a step. The new fixed_mask should now be fix_mask[0, (-2)] = 1. (Since now you've H-1 steps left while rolling out?) And this would change every step?
Hello, let me explain.
I believe the "final state" mentioned by Plankson specifically refers to s_{horizon-1}, rather than the "goal" in a goal-conditioned RL setting. Therefore, if we have a way to continuously indicate the state that needs to be reached after H steps, then the fix_mask would constantly equal 1 on [0, -1].
If you want to implement a planning method that allows the agent to move toward a given "goal" state, you might try modifying the training data to make the last state of trajectories to be generated become the goal, i.e., s_{-1} = goal_state. This way, you can do closed-loop planning using inpainting during inference.
As for the approach you proposed, I believe it's certainly possible, but it requires us to know how many steps are needed to reach the goal state, which isn't very practical.
Thank you very much for the quick reply! Yes indeed at train time we can set the last element as the goal and use a fix_mask of [0,-1].
I didn't really follow how we'd do inpainting at inference time though, at timestep t=0 the fix_mask makes sense. But after taking a step what would our prior and fix_mask be to generate the next action?
Thank you very much for the quick reply! Yes indeed at train time we can set the last element as the goal and use a fix_mask of [0,-1].
I didn't really follow how we'd do inpainting at inference time though, at timestep t=0 the fix_mask makes sense. But after taking a step what would our prior and fix_mask be to generate the next action?
I think each step is the same, like,
at timestep t=0, we observe state $s_0$ from env and get the goal $s_{goal}$ from a certain model, then we set
Thanks again for the detailed reply! But according to this won't the horizon to reaching goal always be H? (That is even after taking a step in the environment, you're still telling the model you've to reach the goal after H more steps)
Thanks again for the detailed reply! But according to this won't the horizon to reaching goal always be H? (That is even after taking a step in the environment, you're still telling the model you've to reach the goal after H more steps)
Yeah, that's true. So if you want to perform planning in this inpainting style, you have to suppose you have a upper model that can tell which state to reach after H steps, like, https://arxiv.org/abs/2401.15443. Otherwise, I think this setting is indeed unreasonable.
Activity
[-]Some doubt problem towards the fix_mask setting in diffusion module[/-][+]Some doubt towards the fix_mask setting in diffusion module[/+]ZibinDong commentedon Jul 11, 2024
Thank you for using CleanDiffuser and providing feedback. : )
Since
fix_maskaffects model training, we cannot change thefix_maskduring inference. Therefore, we do need to retrain the diffusion model with the newfix_mask.Simply try the
fix_maskbelow and train your new diffusion model. : )Plankson commentedon Jul 11, 2024
I see. Thanks for your reply! :)
nkalyanv99 commentedon Jan 2, 2025
Hi, how would you perform closed loop planning at test time though for this in-painting case? Once you generate a trajectory, take a step. The new fixed_mask should now be fix_mask[0, (-2)] = 1. (Since now you've H-1 steps left while rolling out?)
And this would change every step?
ZibinDong commentedon Jan 2, 2025
Hello, let me explain.
I believe the "final state" mentioned by Plankson specifically refers to s_{horizon-1}, rather than the "goal" in a goal-conditioned RL setting. Therefore, if we have a way to continuously indicate the state that needs to be reached after H steps, then the fix_mask would constantly equal 1 on [0, -1].
If you want to implement a planning method that allows the agent to move toward a given "goal" state, you might try modifying the training data to make the last state of trajectories to be generated become the goal, i.e., s_{-1} = goal_state. This way, you can do closed-loop planning using inpainting during inference.
As for the approach you proposed, I believe it's certainly possible, but it requires us to know how many steps are needed to reach the goal state, which isn't very practical.
nkalyanv99 commentedon Jan 2, 2025
Thank you very much for the quick reply! Yes indeed at train time we can set the last element as the goal and use a fix_mask of [0,-1].
I didn't really follow how we'd do inpainting at inference time though, at timestep t=0 the fix_mask makes sense. But after taking a step what would our prior and fix_mask be to generate the next action?
ZibinDong commentedon Jan 3, 2025
I think each step is the same, like,
at timestep t=0, we observe state$s_0$ from $s_{goal}$ from a certain
envand get the goalmodel, then we setwe sample an action to execute
at timestep t=1, we observe state$s_1$ from $s_{goal}$ from a certain
envand get the goalmodel, then we setwe do the same sampling step to get action and execute.
So
fix_masknever changes and we only reset the first element inpriorto tell the model the current observation.nkalyanv99 commentedon Jan 3, 2025
Thanks again for the detailed reply! But according to this won't the horizon to reaching goal always be H? (That is even after taking a step in the environment, you're still telling the model you've to reach the goal after H more steps)
ZibinDong commentedon Jan 3, 2025
Yeah, that's true. So if you want to perform planning in this inpainting style, you have to suppose you have a upper model that can tell which state to reach after H steps, like, https://arxiv.org/abs/2401.15443. Otherwise, I think this setting is indeed unreasonable.
nkalyanv99 commentedon Jan 3, 2025
I see that makes sense. Thanks for clarifying!