Skip to content

Template submission failed #10

New issue
New issue
Closed
Closed
Template submission failed#10

Description

@jdiedrichsen
jdiedrichsen
opened on Mar 10, 2024

Dear Oscar,

I am trying to contribute the SUIT (cerebellar only) template to templateflow, following the submission instructions:
https://www.templateflow.org/contributing/submission/

The file upload does work - however, there seems to be a conflict with something uploaded earlier.

I'm not sure how to resolve this....

best,

Joern

tfmgr add tpl-SUIT --osf-user *******
OSF password: 
Has a RRID (research resource ID) already been assigned? []: No
Preparing Pull-Request (wd=/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74).
Cloning into 'templateflow'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 15 (delta 0), reused 2 (delta 0), pack-reused 13
Receiving objects: 100% (15/15), 44.62 MiB | 20.54 MiB/s, done.
Resolving deltas: 100% (1/1), done.
create(error): /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT (dataset) [collision with content in parent dataset at /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow: ['/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/LICENSE', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/README.md', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/template_description.json', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT.spec', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-image_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-point_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-image_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-point_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_res-1_T1w.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii']]
Traceback (most recent call last):
  File "/opt/miniconda3/bin/tfmgr", line 8, in <module>
    sys.exit(cli())
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/tfmanager/cli.py", line 221, in add
    dl.create(
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad/interface/base.py", line 773, in eval_func
    return return_func(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad/interface/base.py", line 763, in return_func
    results = list(results)
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad_next/patches/interface_utils.py", line 287, in _execute_command_
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'create',
  'message': ('collision with content in parent dataset at %s: %s',
              '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow',
              ['/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/LICENSE',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/README.md',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/template_description.json',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT.spec',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-image_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-point_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-image_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-point_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_res-1_T1w.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii']),
  'path': '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT',
  'refds': None,
  'status': 'error',
  'type': 'dataset'}]

Activity

  1. oesteban commented on Mar 15, 2024

    @oesteban
    oesteban
    on Mar 15, 2024
    Member

    Sorry, sorry, sorry -- I've been up to my eyeballs and totally forgot about this :(

    First, I'm going to move this comment to the template manager. It looks like an error on your computer's side.

    However, I don't want to keep you stuck with this, so if you zip up the template and send it to me, I'll make sure to publish it before Tuesday.

    Again, thanks a lot for your incredible patience (since September, when you first contacted me).

  2. oesteban
    transferred this issue fromtemplateflow/templateflowon Mar 15, 2024

  3. oesteban commented on Mar 15, 2024

    @oesteban
    oesteban
    on Mar 15, 2024
    Member

    Okay, so the installation is failing at

    python-manager/tfmanager/cli.py

    Lines 221 to 226 in f4c596a

    dl.create(
    path=f"tpl-{template_id}",
    cfg_proc="text2git",
    initopts={"initial-branch": "main"},
    description=metadata["Name"],
    )

    meaning, it is trying to create a new datalad dataset but failing. It seems that your templateflow super-dataset at /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow has already a tpl-SUIT installation. I'm not sure why this may be happening, except if you are not using the "official" templateflow super-dataset.

    Something's off, since your error back in September was slightly different:

    jdiedrichsen@diedrichsenofficemac templateflow % tfmgr add tpl-SUIT --osf-user ******
OSF password: 
Has a RRID (research resource ID) already been assigned? []: no
Preparing Pull-Request (wd=/var/folders/sz/7lqgfrln5738xc_hktty2sb00000gp/T/tmpamzeer03).
Cloning into 'templateflow'...
warning: Could not find remote branch tpl-intake to clone.
fatal: Remote branch tpl-intake not found in upstream origin
/bin/sh: hub: command not found
Traceback (most recent call last):
  File "/Users/jdiedrichsen/opt/miniconda3/bin/tfmgr", line 8, in <module>
    sys.exit(cli())
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/tfmanager/cli.py", line 205, in add
    run_command(
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/tfmanager/io.py", line 29, in run_command
    proc = run(cmd, capture_output=capture_output, shell=True, env=env, cwd=cwd,)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/sz/7lqgfrln5738xc_hktty2sb00000gp/T/tmpamzeer03/templateflow'

  4. oesteban commented on Mar 15, 2024

    @oesteban
    oesteban
    on Mar 15, 2024
    Member

    By reading the prompt, it looks like you are executing tfmgr under an existing templateflow repo. If that repo is a fork (I know you did a fork because you invited me to it a while ago), it is possible that git is getting confused.

  5. jdiedrichsen commented on Mar 15, 2024

    @jdiedrichsen
    jdiedrichsen
    on Mar 15, 2024
    Author

    Maybe I am just doing something wrong.

    Here is what I did:

    • I created a new repository called templateflow, only with tpl-SUIT in it- I did NOT fork or clone the main templateflow repro (maybe that's the problem?)
    • I then called tfmgr add tpl-SUIT
    • When I got the first error (in September), I create a branch tpl-intake manually, which avoided the first error, but gave me the second

    I also tried to call tfmgr without having a dedicated repro, but that did not work either.

    Let me know what the correct approach is - and I give it another try from here.

    Joern

  6. jdiedrichsen commented on Apr 11, 2024

    @jdiedrichsen
    jdiedrichsen
    on Apr 11, 2024
    Author

    I tried a couple of more variations, but couldn't make it work - did I have to fork from templateflow?

    A zipped version of the template (46MB) is a bit big to upload here - so here is a dropbox link:

    https://www.dropbox.com/scl/fi/aqqysabl0esns730mzlap/tpl-SUIT.zip?rlkey=rnke2wosbmr0g74g2dthp7lyv&dl=0

    if you could submit from your side, that would be great!

    Joern

  7. oesteban commented on Apr 13, 2024

    @oesteban
    oesteban
    on Apr 13, 2024
    Member

    Hi @jdiedrichsen, I have confirmed that the templateflow manager is quite broken.

    I think your errors come from two assumptions:

    • You will only once ever upload a template to templateflow (okay, fine, this was your first)
    • The root templateflow repository does not change since you fork it the first time. I'm afraid this is what happened in your case (although I will need to investigate it deeper).

    However, even if you had been more lucky and got past the point where it broke for you, then you would've hit several authentication issues. Datalad has changed substantially since I last updated the manager and now things are not working.

    In the end, I had to manually push the submission process. I have taken a few liberties:

    • I removed the transforms files because they were 3D images with a gradient. It looks they could be a 4x4 matrix instead. I wanted to confirm this point.
    • I added two atlases: Diedrichsen2009 and Buckner2011. I hope that is okay with you :)
      My plan is to add the other two or three you have in your repo.

    I just tested and things seem to work:

    • The online browser shows SUIT:
    • The command line lists the files in the template: 
      $ templateflow --version
TemplateFlow Python Client 24.2.0.dev49+gb1f2eb2a.d20240413
$ templateflow update
$ templateflow ls SUIT
/data/templateflow/tpl-SUIT/CHANGES
/data/templateflow/tpl-SUIT/LICENSE
/data/templateflow/tpl-SUIT/README.md
/data/templateflow/tpl-SUIT/template_description.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_dseg.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_stat-confidence_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_T1w.nii.gz
    • And they seem correctly fetched if we request them 
      $ templateflow get SUIT
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.67k/9.67k [00:00<00:00, 3.57MB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51.7k/51.7k [00:00<00:00, 554kB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 776k/776k [00:00<00:00, 1.41MB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.6k/10.6k [00:00<00:00, 3.84MB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44.3k/44.3k [00:00<00:00, 475kB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.77M/1.77M [00:01<00:00, 1.73MB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_flat.surf.gii
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 591k/591k [00:00<00:00, 1.05MB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_sulc.shape.gii
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 147k/147k [00:00<00:00, 534kB/s]
Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_T1w.nii.gz
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385k/385k [00:00<00:00, 728kB/s]
/data/templateflow/tpl-SUIT/CHANGES
/data/templateflow/tpl-SUIT/LICENSE
/data/templateflow/tpl-SUIT/README.md
/data/templateflow/tpl-SUIT/template_description.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_dseg.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_stat-confidence_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.json
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.tsv
/data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
/data/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii
/data/templateflow/tpl-SUIT/tpl-SUIT_T1w.nii.gz

    I'm going to close this issue and open new ones about the submission process. For questions/problems/future topics relating SUIT (e.g., adding the remainder of atlases, or fixing things I didn't get correctly), let's use the template repository at https://github.com/templateflow/tpl-SUIT (I've enabled the issues tab to keep track of these)

    Thanks a lot for your patience :)

  8. oesteban
    closed this as completedon Apr 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions