will there be ablation studies? #13

Activity

1. yulonghui commented on Nov 19, 2023

   

   yulonghui

   on Nov 19, 2023

   Member

   More actions

   Hi, yucc-leon
   Thanks for your suggestions and also thanks for the reference of SkyMath！
   you mentioned that "However have you ever thought about doing ablation study and analysis on the effects by different tasks and finding out why they can improve the performance?"

   What type of ablation experiments are you looking forward to seeing? Feel free to provide details. Maybe we have conducted and can show you, or if we find them equally interesting, we may consider exploring new attempts as well.
2. yucc-leon commented on Nov 20, 2023

   

   yucc-leon

   on Nov 20, 2023

   Author

   More actions

   Hi, yucc-leon 嗨，yucc-leon Thanks for your suggestions and also thanks for the reference of SkyMath！ 感谢您的建议，也感谢SkyMath的参考！ you mentioned that "However have you ever thought about doing ablation study and analysis on the effects by different tasks and finding out why they can improve the performance?" 您提到“但是，您是否考虑过对不同任务的效果进行消融研究和分析，并找出为什么它们可以提高性能？“

   What type of ablation experiments are you looking forward to seeing? Feel free to provide details. Maybe we have conducted and can show you, or if we find them equally interesting, we may consider exploring new attempts as well.你期待看到什么样的消融实验？请随意提供细节。也许我们已经进行了，可以给你看，或者如果我们发现它们同样有趣，我们也可以考虑探索新的尝试。

   Thanks for the replying and opensourced datasets recently and we can see more details:)
   I came up with such requests because Skywork said their work was very similar to MetaMath and other related researches. That meant they also made up some augmented datasets using some secret prompt engineering, and their datasets were more suitable to train LLMs for better scores in GSK8K and MATH.
   So I guess, if one can compare their datasets with yours or do some ablation studies like using only one or two of your datasets to find out what kind of task or task combinations makes the main contribution of the final performance, the community may be closer to the answer to know how to make up the best (or very close to it) datasets to train a well-performed LLM for specific tasks.