Skip to content

Example script for continued pre-training? #162

Open

Description

@jphme

First of all many thanks for the release of llama 2 7b 32k and your precious contributions!

It's appreciated that you provide example scripts for Finetuning; however the (for me) most interesting part, the continued pre-training mentioned in the blog post, is missing.

Would it be possible to provide that script as well? Many thanks in advance+ all the best!

Activity

  1. lllyyyqqq commented on Aug 3, 2023

    @lllyyyqqq

    Hi, I am also interested in the pre-training part. A script would be the best. Also, a quick question, what's the sequence length setting in the pre-training? 4k or 32k?

  2. csris commented on Aug 6, 2023

    @csris
    Contributor

    @zhangce @LorrinWWW, can you please comment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions