Is the MetaMathQA dataset suitable for pre-training? #22

Activity

1. yulonghui commented on Dec 21, 2023

   

   yulonghui

   on Dec 21, 2023

   Member

   More actions

   Hi，Jingyi, Many thanks for your kind words.
   I think MetaMathQA is suitable for pre-training. In fact, OpenChat-3.5 utilized our data and indeed achieved impressive performance. https://huggingface.co/openchat/openchat_3.5

   However, one thing I believe needs attention is that you should publicly disclose the type of data you used, such as MetaMathQA. This way, others can assess whether your model has encountered GSM8K-like data.