Updating bag-info.txt makes bag invalid

When a bag has a tag manifest file created using an algorithm other than md5, such as the following bag:

bag-info.txt         
bagit.txt            
data                 
manifest-sha1.txt    
tagmanifest-sha1.txt

Updating bag info results in an invalid bag:

>>> import bagit
>>> bag = bagit.Bag('/Users/twan/edeposit/edeposit/apps/signiant/tests/SR1-4444_20140611200739')
>>> bag.is_valid()
True
>>> bag.info['External-Identifier']='xyz'
>>> bag.save()
>>> bag.is_valid()
False

The problem looks like it assumes the tag manifest file always uses md5.

# Update tag-manifest for changes to manifest & bag-info files
        _make_tagmanifest_file('tagmanifest-md5.txt', self.path)

added test for LibraryOfCongress#32, which passes?

@tongwang

@tongwang I added a test case that (I thought) would exercise the problem, but it passes. Can you get it to fail? Also, what do you see as output when you run validate()?

@edsu

@edsu the issue is that in the test case even you create the bag with "sha1", the tag manifest file is still in md5. It is hardwired to md5:

def _make_tagmanifest_file(tagmanifest_file, bag_dir):
    files = [f for f in listdir(bag_dir) if isfile(join(bag_dir, f))]
    checksums = []
    for f in files:
        if f == tagmanifest_file:
            continue
        fh = open(join(bag_dir, f), 'rb')
        m = hashlib.md5()
        while True:
            bytes = fh.read(16384)
            if not bytes:
                break
            m.update(bytes)
        checksums.append((m.hexdigest(), f))
        fh.close()

    with open(join(bag_dir, tagmanifest_file), 'w') as tagmanifest:
        for digest, filename in checksums:
            tagmanifest.write('%s %s\n' % (digest, filename))

Ouch! I thought there were tests for sha1 manifest generation. I'll check.

It does seem wrong that it would always write a tagmanifest-md5.txt even if you have a manifest-sha1.txt ; but why does this make the bag invalid? Can you construct a test case that fails?

Oh, in your example you probably have a bag created with another tool which has a tagmanifest-sha1.txt that is not being updated?

Ok, I've got a test case that is failing now over here.

Yes, the existing tagmanifest-sha1.txt is not updated, which fails validation.

allow non-md5 tagmanifests to be created. fixes LibraryOfCongress#32

@tongwang

@tongwang can you give my latest branch a try? https://github.com/edsu/bagit/tree/issue32 I believe I fixed the issue.

@edsu

@edsu it works great! Thank you for the quick fix.

Awesome. I'll send a pull request, maybe you could merge it in (since I no longer work at LC)?

Navigation Menu

Repository navigation

Updating bag-info.txt makes bag invalid #32

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Updating bag-info.txt makes bag invalid #32

Description

Activity

edsu commented on Sep 19, 2014

tongwang commented on Sep 19, 2014

edsu commented on Sep 19, 2014

edsu commented on Sep 19, 2014

edsu commented on Sep 19, 2014

edsu commented on Sep 19, 2014

tongwang commented on Sep 19, 2014

edsu commented on Sep 19, 2014

tongwang commented on Sep 19, 2014

edsu commented on Sep 19, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions