You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for sharing the code of the wonderful work. I have a problem with the loss function. In section 2.2, it introduces the loss function of knowledge distillation, which is used to obtain the high entropy. In section 2.3, it introduces the loss function of uncertainty minimization, which is used to obtain the low entropy. Will there be a conflict between these two loss functions? I hope to receive your reply as soon as possible.
The experimental results demonstrate that these two loss functions are not contradictory. Regarding the issue you mentioned, I believe that the loss function based on Knowledge Distillation (KD) does not increase entropy.
In section 2.2, you said "Note that T = 1 corresponds to a standard
Softmax function, and a larger T value leads to a softer probability distribution
with higher entropy. ". In the experiment, you set T=10. Could you explain how the knowledge distillation works in principle? (rather than just answering this question from the perspective of experimental results) Thanks.
This entropy refers to an increase in the entropy of the softened
distribution, but it doesn't necessarily imply that the goal of this loss
is to increase the entropy of the prediction results.
abcxubu ***@***.***> 于2023年8月25日周五 16:42写道:
In section 2.2, you said "Note that T = 1 corresponds to a standard
Softmax function, and a larger T value leads to a softer probability
distribution
with higher entropy. ". In the experiment, you set T=10. Could you explain
how the knowledge distillation works in principle? (rather than just
answering this question from the perspective of experimental results)
Thanks.
—
Reply to this email directly, view it on GitHub
<#2 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWZLYV33EPQEHLCVKTG2LWDXXBQO7ANCNFSM6AAAAAA3ZLUFEQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
Activity
lanfz2000 commentedon Aug 24, 2023
The experimental results demonstrate that these two loss functions are not contradictory. Regarding the issue you mentioned, I believe that the loss function based on Knowledge Distillation (KD) does not increase entropy.
abcxubu commentedon Aug 25, 2023
In section 2.2, you said "Note that T = 1 corresponds to a standard
Softmax function, and a larger T value leads to a softer probability distribution
with higher entropy. ". In the experiment, you set T=10. Could you explain how the knowledge distillation works in principle? (rather than just answering this question from the perspective of experimental results) Thanks.
lanfz2000 commentedon Aug 25, 2023