Skip to content

use file.copy instead of file.rename #9

Open
Open
use file.copy instead of file.rename#9
Assignees
lshep

Description

@LiNk-NY
LiNk-NY
opened on Apr 19, 2018

Hi Lori, @lshep

I got this error message when copying a file from my /home/$USER/Downloads to
another partition on our server.

In file.rename(fpath, rpath) :
  cannot rename file '/home/mramos/Downloads/hnsc_tcga.tar.gz' to '/data/16tb/cbio/2e7912cc3e62_hnsc_tcga.tar.gz', reason 'Invalid cross-device link'

This has been seen before in vtest issue 14
and it looks like the solution is to use file.copy.

Could this be changed in BiocFileCache?
Thanks!

Regards,
Marcel

Activity

  1. lshep
    self-assigned this
    on Apr 19, 2018

  2. lshep commented on Apr 19, 2018

    @lshep
    lshep
    on Apr 19, 2018
    Contributor

    Can you give me an example of the code you are trying to run that causes this?

  3. mtmorgan commented on Apr 19, 2018

    @mtmorgan
    mtmorgan
    on Apr 19, 2018
    Contributor

    one consequence of this is that it, well, copies the file, which could be very expensive; maybe try file.rename and on failure file.copy? Or determine a priori whether file.copy is need (separate file systems; not sure how one would do that...)

  4. LiNk-NY commented on Apr 19, 2018

    @LiNk-NY
    LiNk-NY
    on Apr 19, 2018
    ContributorAuthor

    Hi Lori, @lshep

    Here is some code to reproduce the issue.
    This would require that you have a separate partition installed on your system or a USB drive.

    in a shell

    cd ~/Downloads && curl -O http://download.cbioportal.org/coadread_mskcc.tar.gz

    in R

    library(BiocFileCache)
bfc <- BiocFileCache("/media/lshep/USB")
bfcadd(bfc, "coadread_mskcc", fpath = "~/Downloads/coadread_mskcc.tar.gz", download = FALSE, action = "move")

    @mtmorgan Using file.copy when the rename fails sounds like a good alternative.

    Thanks,
    Marcel

  5. lshep commented on Apr 19, 2018

    @lshep
    lshep
    on Apr 19, 2018
    Contributor

    And why couldn't you create the cache in the partition where the files are?

    You can always use exportbfc to make the bfc and selection of files portable to put on a drive

  6. LiNk-NY commented on Apr 19, 2018

    @LiNk-NY
    LiNk-NY
    on Apr 19, 2018
    ContributorAuthor

    I can create the cache there but I can't move a file across partitions using bfcadd or file.rename (internally).

  7. LiNk-NY commented on Apr 19, 2018

    @LiNk-NY
    LiNk-NY
    on Apr 19, 2018
    ContributorAuthor

    @lshep
    The USB drive example was just an easy way to reproduce the issue.
    We are actually using a 16 TB partition for storing data. The original code creates the cache in
    this partition without a problem.

    Thanks!

  8. lshep commented on Apr 20, 2018

    @lshep
    lshep
    on Apr 20, 2018
    Contributor

    Just on a quick test I don't think file.copy works either - I think this is a limitation of R and the system:

    > temp = tempfile()
> temp
[1] "/tmp/RtmpJn1pnt/file238138cf692d"
> file.rename(temp, "/home/lori/loristempfile")
[1] FALSE
Warning message:
In file.rename(temp, "/home/lori/loristempfile") :
  cannot rename file '/tmp/RtmpJn1pnt/file238138cf692d' to '/home/lori/loristempfile', reason 'Invalid cross-device link'
> file.copy(temp, "/home/lori/loristempfile")
[1] FALSE

  9. LiNk-NY commented on Apr 27, 2018

    @LiNk-NY
    LiNk-NY
    on Apr 27, 2018
    ContributorAuthor

    Hi Lori, @lshep

    tempfile doesn't create a file, it just gives you a possible name.
    I'm not sure how you're getting that error. I've tried to reproduce but
    I don't get the same warning.

    > temp <- tempfile()
> temp
[1] "/tmp/RtmpTVASgz/file1c963193eee2"
> file.rename(temp, "~/tempfile")
[1] FALSE
Warning message:
In file.rename(temp, "~/tempfile") :
  cannot rename file '/tmp/RtmpTVASgz/file1c963193eee2' to '/home/mr148/tempfile', reason 'No such file or directory'

    If no file exists, you can't copy or rename it.

  10. lshep commented on Apr 27, 2018

    @lshep
    lshep
    on Apr 27, 2018
    Contributor

    Ah yes you are right. But I'm still not quite convinced that this is a good idea. As it has the potential to pollute two different different user locations -
    I guess I don't understand why you can't change the action in bfcadd to action="copy" which would user file.copy instead of file.rename. Then instead of a user potentially polluting their system unintentionally - at least it is intentional

  11. LiNk-NY commented on Apr 27, 2018

    @LiNk-NY
    LiNk-NY
    on Apr 27, 2018
    ContributorAuthor

    This is mostly an edge case for users who have separate partitions in their file systems
    and don't cache in their home folder.

    Yes, I can use action="copy" but I prefer "move" for the same reason one would
    use mv instead of cp. I'm caching large files and I don't want 2 copies of them.

    I liked Martin's idea of using file.copy when file.rename fails and additionally
    removing the original file after the copy for action="move".

    I've used action = "copy" in the meantime and deleted the files.

    Thanks @lshep.

  12. HenrikBengtsson commented on Jan 6, 2025

    @HenrikBengtsson
    HenrikBengtsson
    on Jan 6, 2025

    This is mostly an edge case for users who have separate partitions in their file systems
    and don't cache in their home folder.

    I don't think this is an edge case. On environments with a shared global file system (e.g. HPC environment), working toward local disk can be several orders of magnitude faster (sic!) than working towards the global file system. This is why many of these systems set TMPDIR to point to a local disk on the current machine. For the same reason, one might want to host cache files on a local disk (e.g. under TMPDIR) rather than on the global file system. This can be achieved by setting R_USER_CACHE_DIR or XDG_CACHE_HOME, which tools::R_user_dir(..., which = "cache") respects.

    I liked Martin's idea of using file.copy when file.rename fails and additionally removing the original file after the copy for action="move".

    I second this. Maybe the default could be action = c("move", "copy") to indicate attempts are made in that order, i.e. "copy" is a fallback to "move".

  13. hpages commented on Jan 8, 2025

    @hpages
    hpages
    on Jan 8, 2025
    Contributor

    The fundamental problem is that file.rename() mimics the behavior of the rename Unix system call which is a low-level operation that only works to rename files within a given file system.

    Unix command mv is higher-level and works across partitions. It actually does exactly what Martin is suggesting: When used to move files from one file system to another, it copies the files, then delete the originals. This is of course much more expensive than moving files within a file system where no copies need to be made.

    Has someone ever requested a file.move() function to be added to base R on R-devel? Otherwise I'd be surpised if no package has implemented a file.move()-like function that does exactly that. @HenrikBengtsson Maybe something for R.utils?

  14. scottkosty
    added a commit that references this issue on Jul 13, 2025

    Robust "renaming" across different file systems

    d710295
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions