[Errno24] Too many open files:

[JonathanHungerland]

I am adding lots of single files to my local university's data repository using the python-dvuploader. At some point I always run into the above error. I am circumventing this problem by performing my upload in multiple batches and then closing all open files with psutil after each batch. See the attached python script for the workaround.

I don't know where the error is specifically, but I didn't find a related issue (closed or open) so I imagine this kludge might be helpful for someone else.

upload_to_dataverse.py

[JonathanHungerland]

Oh and it's probably not good practive to attached executable python code to an issue. So here is the core script:

BATCH_SIZE = 160
def check_open_files():
    process = psutil.Process(os.getpid())
    return process.open_files()

#Process files in batches to avoid "too many open files"
for i in range(0, len(all_files), BATCH_SIZE):
    batch = all_files[i:i + BATCH_SIZE]
    print(f"Uploading batch {i // BATCH_SIZE + 1} of {len(all_files) // BATCH_SIZE + 1}")

    dvuploader = dv.DVUploader(files=batch)

    try:
        dvuploader.upload(
            api_token=API_TOKEN,
            dataverse_url=DV_URL,
            persistent_id=PID,
            n_parallel_uploads=PARALLEL_UPLOADS
        )
    except Exception as E:
        print(f"Error in batch {i // BATCH_SIZE + 1}: {E}")
        
    # Manually close open file descriptors
    open_files = check_open_files()
    for file in open_files:
        os.close(file.fd)  # Close file descriptor

I believe this issue is likely related to some on the dataverse side:
IQSS/dataverse#6723
IQSS/dataverse#6656

[JR-1991]

@JonathanHungerland thanks for bringing this up! I think the issue is Python-related, since there is a limit on how many files can be opened at the same time. A possible solution would be either to batch, as you did, or internally open files only when scheduled for an upload. I will take a deeper look into this and check out how this can be solved.

[JR-1991]

@JonathanHungerland, I’ve migrated the code base to open files only when required and have opened a draft PR #45. This should resolve the issue since file handlers are only present when needed, given that the File has been initialized using a path.

Would you mind test PR #45 and let me know if everything worked out?