Indexing tools unable to handle large MAF files #8

Activity

1. jxtx commented on Jul 29, 2016

   

   jxtx

   on Jul 29, 2016

   Contributor

   More actions

   I think pushing the version is the right solution. Somebody just needs to
   find time to work on this...

   -- jt

   On Fri, Jul 29, 2016 at 12:41 PM, Daniel Blankenberg <
   notifications@github.com> wrote:

   One example is the hg38 projected chr1 multiz100way from UCSC:
   http://hgdownload.soe.ucsc.edu/goldenPath/hg38/multiz100way/maf/chr1.maf.gz
   (5.9G gz'd, 66G gunzip'd)

   maf_build_index.py chr1.maf
   Traceback (most recent call last):
   File "/path_to/bin/maf_build_index.py", line 83, in
   if name == "main": main()
   File "/path_to/bin/maf_build_index.py", line 80, in main
   indexes.write( out )
   File "/path_to/lib/python2.7/site-packages/bx/interval_index_file.py", line 332, in write
   write_packed( f, ">I", base )
   File "/path_to/lib/python2.7/site-packages/bx/interval_index_file.py", line 463, in write_packed
   f.write( pack( pattern, *vals ) )
   struct.error: 'I' format requires 0 <= number <= 4294967295

   One possibility is to up the version number and store unsigned integers as
   unsigned long long >Q, which would max out at 18446744073709551615 vs
   4294967295. Would double the packed size though.

   Another potential workaround could be to break the MAF up into multiple
   files, but I haven't tested this.

   xref: https://biostar.usegalaxy.org/p/18196/

   —
   You are receiving this because you are subscribed to this thread.
   Reply to this email directly, view it on GitHub
   #8, or mute the thread
   https://github.com/notifications/unsubscribe-auth/AAE4ZSWKFE4vERJLatl6XcOwXGIzEhg3ks5qai1QgaJpZM4JYXhp
   .
2. poojanarang commented on Aug 1, 2016

   

   poojanarang

   on Aug 1, 2016

   More actions

   Thanks for looking into this.
   This issue have been bothering us for several weeks. Hope to have a solution to this problem soon.
3. jodyhey commented on Mar 22, 2021

   

   jodyhey

   on Mar 22, 2021

   More actions

   Any progress on this?
   I just ran into what seems to be the same problem on a large MAF file (48Gb):

   (base) /mnt/e/genemod/better_dNdS_models/drosophila/11_6_2020/cactus_work$ python /home/jodyhey/miniconda3/bin/maf_build_index.py drosophila_cactus.maf drosophila_cactus.mafindex

   Traceback (most recent call last):
   File "/home/jodyhey/miniconda3/bin/maf_build_index.py", line 82, in
   main()
   File "/home/jodyhey/miniconda3/bin/maf_build_index.py", line 77, in main
   indexes.write(out)
   File "/home/jodyhey/miniconda3/lib/python3.8/site-packages/bx/interval_index_file.py", line 351, in write
   write_packed(f, ">I", base)
   File "/home/jodyhey/miniconda3/lib/python3.8/site-packages/bx/interval_index_file.py", line 486, in write_packed
   f.write(pack(pattern, *vals))
   struct.error: 'I' format requires 0 <= number <= 4294967295
4. nsoranzo commented on Mar 22, 2021

   

   nsoranzo

   on Mar 22, 2021

   Collaborator

   More actions

   @jodyhey No one is working on this issue, sorry, but pull requests are welcome!
5. pkncsk commented on Mar 24, 2022

   

   pkncsk

   on Mar 24, 2022

   Last edited by pkncsk

   More actions

   Is there any update on this issue? with how .maf files have gotten bigger lately, this might become a more common issue.

   here I ran the script on a 30 Gb .lzo file (85 Gb uncompressed)

   python3 maf_index.py
   
   Traceback (most recent call last):
     File "maf_index.py", line 75, in <module>
       main()
     File "maf_index.py", line 70, in main
       indexes.write(out)
     File "/home/pc575/jupyter-env-icelake/lib/python3.7/site-packages/bx/interval_index_file.py", line 351, in write
       write_packed(f, ">I", base)
     File "/home/pc575/jupyter-env-icelake/lib/python3.7/site-packages/bx/interval_index_file.py", line 486, in write_packed
       f.write(pack(pattern, *vals))
   struct.error: 'I' format requires 0 <= number <= 4294967295