Dataset.list_files() returns None (?)

[kbrueckmann]

Dataverse: v. 6.3 build v6.3+10797_dls
easyDataverse: f4518a0
Python: 3.12.3

The function list_files() returns None for me:

        dataverse = Dataverse(
            server_url="https://heidata.uni-heidelberg.de/"
        )
   
        dataset = dataverse.load_dataset(
            pid=pid,
            download_files=False
        )

        files = dataset.list_files()

Tested with pid values "doi:10.11588/data/JRWPFV", "doi:10.11588/data/DVU14P" and others of our instance.

[JR-1991]

@kbrueckmann, thanks for submitting the issue! This is related to setting download_files to False, which does not fetch files and metadata. At the time of implementation, the intent was to prevent accidentally updating files if a similar-named file was present. However, this may be an unlikely edge case, but it could be quite dangerous if it happens.

I could think of a solution, that fetches file metadata and "locks" upload. This means that the metadata is present and can be changed, but no file updates are possible. Do you think this would be helpful?

[kbrueckmann]

@JR-1991 Thank you for the offer! My use case was a script that would check locally saved files against a dataset to either add them (if not present already), update them (if present) or skip them (e.g. in case of same content, invalid characters in the file name etc.) because we received quite a few tickets from data contributors wishing to update their data. The update takes place when path and file name of the remote and local file are the same, so we're using that on purpose; but I definitely do see the risks for data contributions where this isn't intended. As I did switch to another library by now I see no problem in closing the issue if others do not need this functionality.

[JR-1991]

@kbrueckmann, thanks for specifying your use case! If I understood correctly, you want to download specific files of a dataset and update them accordingly.

This is possible by leaving download_files to True and specifying a list of files you want to download/update. The filenames parameter can be used to set these, which will download only files matching these paths (regex is also possible).

Given the file(s) exist, easyDataverse tracks changes and only uploads/updates the file if its hash has changed. I have set up a small Colab notebook to demonstrate how it can be done:

https://colab.research.google.com/drive/1h8M4rwxoqwn8h4Poe31ytkj7e1iG8Ueb?usp=sharing

Does this reflect your use case?