Skip to content

Fix Schirrmeister2017 re-downloading every recording - #1132

Open
adityasingh2400 wants to merge 1 commit into
NeuroTechX:developfrom
adityasingh2400:fix/schirrmeister2017-redownload
Open

Fix Schirrmeister2017 re-downloading every recording#1132
adityasingh2400 wants to merge 1 commit into
NeuroTechX:developfrom
adityasingh2400:fix/schirrmeister2017-redownload

Conversation

@adityasingh2400

Copy link
Copy Markdown

Schirrmeister2017.data_path downloads each subject's train and test EDF through moabb.datasets.download.data_dl, then moves the file out of the directory data_dl owns (MNE-schirrmeister2017-data/robintibor/high-gamma-dataset/raw/master/data/<train|test>/) and into MNE-schirrmeister2017-data/<train|test>/. data_dl derives its destination from the URL, so on the next call it looks in the directory the file was moved out of, finds nothing, and fetches the whole recording again. The refetched copy is then abandoned where it landed, because the move is guarded by if not os.path.exists(new_path) and the relocated file is already there. The net effect is one redundant full re-download of every file plus two permanent copies of a multi-gigabyte recording on disk. This is what @toncho11 reported in #851.

The relocation was cosmetic. data_dl already keeps the train and test files in separate directories, since it mirrors the URL path, so the extra train/ and test/ folders were not adding any separation that was missing.

data_path now returns the path data_dl reports and never writes to the old location. It still reads from it first, so anyone with data already on disk from a previous MOABB version keeps using that copy instead of downloading ~50 GB again. With force_update=True the stale relocated file is removed before the refresh, otherwise it would shadow the newly downloaded one on the following call.

Tested with two regression tests in moabb/tests/test_dataset_fixes.py that stub data_dl with a stand-in reproducing its caching contract, so no network access is involved. test_schirrmeister2017_does_not_redownload calls data_path three times and asserts two downloads and two files on disk (on develop it observes four downloads and four files). test_schirrmeister2017_reuses_relocated_files pre-populates the old layout and fails if anything is downloaded at all. Both fail before the change and pass after it, and pre-commit run --files is clean on the three touched files.

Fixes #851

data_path moved each freshly downloaded EDF out of the directory owned by
data_dl and into MNE-schirrmeister2017-data/<train|test>/. That left the
download cache empty, so the next call fetched the whole multi-gigabyte
recording again, and the refetched copy was then abandoned in place because
the destination already existed.

data_path now returns the path reported by data_dl and only reads from the
old layout when a file is already there, so an existing local copy is still
reused and never downloaded twice.

Fixes NeuroTechX#851
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problems with datasets downloads

1 participant