Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Not able to use get_directory_contents #10

Open

Description

@jbd

Hello,

I'm trying to list a directory with 718 directories in it using get_directory_content API function, but I can't make it work.

If I don't use a limit option, or if I use a limit option higher than the number of directories I've got, I can't
get anything from the function. AM I misusing something ?

api_response = ns_api.get_directory_contents(options.path) # does not work
print(api_response) # {'children': None, 'resume': None}

api_response = ns_api.get_directory_contents(options.path, limit=8192) # does not work
print(api_response) # {'children': None, 'resume': None}
api_response = ns_api.get_directory_contents(options.path, limit=1) # does work
print(api_response) # print {'children': [{'access_time': None,
                             #  'atime_val': None,
                             # 'block_size': None,
                             # 'blocks': None,

And even with limit=1, I'm not able to resume the listing after with something like that:

api_response = api.get_directory_contents(directory, detail='name,gid', type='container', limit=1)
            
while api_response.resume:
                api_response = api.get_directory_contents(directory, resume=api_response.resume)
                for container in api_response.children:
                        yield container

Here is a usable gist: https://gist.github.com/jbd/8a5f691e75509dc5397243a105dc61fa

Activity

  1. Atomicology commented on Oct 31, 2018

    @Atomicology

    @jbd: The options.path needs to be prefixed with ifs, so if you are looking to query the directory /ifs/data, then it must be specified as follows:

    api_response = ns_api.get_directory_contents('ifs/data')
    

    This method is unfortunately not flexible enough to support either an absolute path with / or a relative path under /ifs/ because these language bindings are auto-generated. Thus, if your path is absolute, then you will need to pass it to get_directory_contents as options.path[1:]. Otherwise, if it's relative to /ifs, then you will need to prepend it as 'ifs/' + options.path. Hopefully that helps.

  2. jbd commented on Nov 1, 2018

    @jbd
    Author

    Hi,

    thank you for your answer. I guess my initial message was not clear enough. I've got no problem with the directory_path subtelties but I don't seem to be able to use get_directory_contents correctly.

    Here is a full example, I hope it will be easier to follow.

    1. Create a thousand directories on the Isilon cluster
    ISILON-1# mkdir -p /ifs/data/apitest/{0000..1023}
    ISILON-1# find  /ifs/data/apitest -type d |wc -l
        1025
    

    I've got no problem to make get_directory_contents to work with this code :

    if limit:
        api_response = api_instance.get_directory_contents(directory_path, limit=limit)
    else:
        api_response = api_instance.get_directory_contents(directory_path)
    
    if api_response.children:
        pprint(len(api_response.children))
    else:
        pprint(api_response)
    

    See https://gist.github.com/jbd/3504a32fdc3be782d774198e55711ea4 for the full usable program

    $ python issue_simple.py ifs/data/apitest
    1000
    

    This corresponds to default value of the "limit" parameter.

    The rest of the tests looks OK:

    128
    $ python issue_simple.py ifs/data/apitest 256
    256
    $ python issue_simple.py ifs/data/apitest 512
    512
    $ python issue_simple.py ifs/data/apitest 4096
    1024
    

    But if I use another directory with some contents, api_response.children is None :

    ISILON-1# find /ifs/projets/p01 -maxdepth 1 | wc -l
         719
    
    ISILON-1# ls -led /ifs/projets/p01
    drwxr-xr-x    719 root  wheel  19782 Oct 23 12:15 /ifs/projets/p01
     OWNER: user:root
     GROUP: group:wheel
     SYNTHETIC ACL
     0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child 
     1: group:wheel allow dir_gen_read,dir_gen_execute 
     2: everyone allow dir_gen_read,dir_gen_execute
    
    $ python issue_simple.py ifs/projets/p01
    {'children': None, 'resume': None}
    

    If I specify an arbitrary limit in the get_directory_contents call, it works until a certain size:

    $ python issue_simple.py ifs/projets/p01 128
    128
    $ python issue_simple.py ifs/projets/p01 256
    256
    $ python issue_simple.py ifs/projets/p01 512
    {'children': None, 'resume': None}
    

    The sweet spot is limit=318:

    $ python issue_simple.py ifs/projets/p01 318
    318
    $ python issue_simple.py ifs/projets/p01 319
    {'children': None, 'resume': None}
    

    What could explain this behavior ? The equivalent curl commands with the same user are working fine.

    $ curl --referer XXXX -H 'Accept: application/json; indent=4' -H 'X-CSRF-Token: 14829322-bcbd-4889-89f7-XXXXXXXXX' -s --insecure -b @cookie https://isiloncluster:8080/namespace/ifs/projets/p01 | wc -l
    718
    
  3. jbd commented on Nov 1, 2018

    @jbd
    Author

    I've got a directory with a '\303' C escape code in it. This is a problem, but I guess this should not make get_directory_contents working incorrectly (or it should raise an exception) and silently fails.

    Here is a reproducer:

    ISILON-1# mkdir -p /ifs/data/apitest/$(printf 'weird\303_directory')
    ISILON-1# ls /ifs/data/apitest 
    weird_directory
    ISILON-1# ls -b /ifs/data/apitest
    weird\303_directory
    
    $ python issue_simple.py ifs/data/apitest 
    {'children': None, 'resume': None}
    

    The curl command is working:

    $ curl --referer XXXXX -H 'Accept: application/json; indent=4' -H 'X-CSRF-Token: 8655b8e1-cc68-42dc-8b15-XXXXX' -s --insecure -b @cookie https://isiloncluster:8080/namespace/ifs/data/apitest
    {"children":[{
       "name" : "weird_directory"
    }
    
  4. jbd commented on Nov 1, 2018

    @jbd
    Author

    In the case of the "weird_directory" with the embedded non-printable char, the response.data object is not considered as a valid json here:

    data = json.loads(response.data)

    The fact that it cannot be transformed to json seems like the kwargs cannot be build in deserialize_object:

    isinstance(data, (list, dict))):

  5. added
    Bug_SDKBugs related to SDK and surrounding scripts
    and removed on Apr 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Bug_SDKBugs related to SDK and surrounding scripts

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions