Exceeding these limits will trigger Input/output error as above, and show a notification in the Colab UI. Google Drive enforces various limits, including per-user and per-file operation count and bandwidth quotas. Why do Drive operations sometimes fail due to quota? The link of the error message points to a Google Collaboratory page, more specifically to the following explanation: Google Drive file size limit is 5TB, so it doesn't look to be a problem solely related to the file size trying again and again, you are just delaying when you can upload your file. Now I am not so sure its as simple as just running the code as I am stuck in the first part only, give me some ways to get around this issue. Furthermore after I get these video clips, I have to preprocess them on colab itself and store the final preprocessed data (120GB) to the drive. What is the problem? How do people work with large datasets on colab. I am using my college email id so there is TBs of drive space available and the internet connection is also good, I have tried this many times but at some point, this pops up and the data doesn't go into drive. The whole data that's being unzipped wont cross 100GB. I am able to redo the process again and again but it works initially but then this pop up shows up after 2-3 minutes. The video clips were being unzipped but after 2-3 minutes they were not being uploaded to drive anymore, while the code is running a pop up shows up:Ī Google Drive quota has been exceeded (most recently at 12:29:18 AM). Now I am directly trying to untar it to same folder in the drive as follows: !tar -xvf 'video_clips.tar' So I thought I will directly download to Drive through Colab, I downloaded 20 GB tar file from the source using wget directly to Drive using the following: from lab import driveĪll is well and good till here.

If I upload it from my PC its taking a hell lot of time. I need to store a dataset (120GB) in my drive to later train a model on it.

Download google drive 750gb