Automate downloading Kaggle Datasets using Kaggle API

Manually downloading datasets from Kaggle every time can quickly become repetitive. A better approach is to use the Kaggle API to download competition datasets automatically with simple code. This method creates a repeatable workflow, keeps your project organized, and makes it easier to reproduce experiments. In this tutorial, you’ll learn how to configure the Kaggle API, authenticate your account, and download competition files using Python.

Prerequisites

Before you begin, make sure you have:

• A Kaggle account
• A Kaggle API token (kaggle.json)
• The Kaggle Python package installed
• A Jupyter Notebook
• A project folder where your datasets will be stored

Step 1: Generate Your Kaggle API Token

Log in to your Kaggle account and navigate to Account Settings.

Under the API section, click Create New API Token.

Kaggle downloads a file named:

kaggle.json

This file contains your Kaggle username and API key, allowing Python scripts and notebooks to authenticate with Kaggle.

Step 2: Save kaggle.json in the Correct Location

Move the downloaded file into the Kaggle configuration folder.

On Windows:

C:\Users\YourUsername\.kaggle\kaggle.json

If the .kaggle folder doesn’t exist, create it manually.

Once the file is in place, the Kaggle CLI can authenticate automatically.

Step 3: Install the Kaggle Package

Install the Kaggle library using either pip or conda.

pip install kaggle

or

conda install kaggle

After installation, the Kaggle command-line interface (CLI) becomes available inside Jupyter Notebook.

Step 4: Organize Your Project Structure

A clean project structure makes machine learning projects much easier to maintain.

A common layout looks like this:

project/
├── data/
        │  
        └── raw/
        └── notebook/    
                └── kaggle_download.ipynb

In this structure:

• data/raw stores the original downloaded files.
• notebook contains your Jupyter notebooks.

Separating code from raw data is considered a best practice for data science projects.

Step 5: Open Your Jupyter Notebook

Launch Jupyter Notebook and open your project notebook.

Inside notebook cells, shell commands can be executed by placing an exclamation mark (!) before the command.

For example:

!kaggle --help

This tells Jupyter to execute the command in the system shell instead of Python.

Step 6: Create a Function to Download Competition Files

Rather than writing the download command repeatedly, create a reusable Python function.

def extract_data(file_name, file_path):    !kaggle competitions download -c titanic -f {file_name} -p {file_path} --force

The command works as follows:

Using a function makes it easy to reuse the same code for different competitions.

Step 7: Create the Download Path

Use Python’s os module to build paths that work regardless of your current working directory.

import os 
raw_data_path = os.path.join(os.pardir, "data", "raw")

This points to:

project/data/raw

Building paths programmatically makes your notebook more portable across different operating systems.

Step 8: Define the Dataset File Names

Store the competition file names in variables.

train_file_name = "train.csv"
test_file_name = "test.csv"

Using variables keeps your code readable and makes changing file names much easier.

Step 9: Download the Kaggle Competition Dataset

Now call the function for each dataset.

extract_data(train_file_name, raw_data_path)
extract_data(test_file_name, raw_data_path)

Once complete, the folder should contain:

data/
   └── raw/    
       ├── train.csv   
       └── test.csv

The datasets are now ready for preprocessing and model training.

Complete Example

import os
def extract_data(file_name, file_path):   
  !kaggle competitions download -c titanic -f {file_name} -p {file_path} --force
  
  
raw_data_path = os.path.join(os.pardir, "data", "raw")
train_file_name = "train.csv"
test_file_name = "test.csv"
extract_data(train_file_name, raw_data_path)
extract_data(test_file_name, raw_data_path)

Why Use the Kaggle API in Jupyter Notebook?

Using the Kaggle API in Jupyter Notebook offers several advantages:

• Eliminates repetitive manual downloads
• Creates reproducible machine learning workflows
• Keeps datasets organized in a consistent folder structure
• Easily integrates into automated data pipelines
• Makes collaboration easier because setup steps can be scripted

For larger projects, automating dataset downloads can save significant time.

Common Mistakes to Avoid

If your download isn’t working, check the following:

• Ensure kaggle.json is in the correct folder.
• Install the Kaggle package before running commands.
• Use the correct competition name after the -c flag.
• Verify the filename after the -f flag.
• Make sure the destination directory already exists.
• Confirm you’ve accepted the competition rules on Kaggle. Many competition datasets require acceptance before API downloads are permitted.

Conclusion

Using the Kaggle API in Jupyter Notebook is one of the easiest ways to automate downloading competition datasets. Once you’ve configured your API credentials, downloading files becomes a simple one-line command that can be reused across projects.

Whether you’re working on the Titanic competition or any other Kaggle challenge, this approach improves reproducibility, keeps your project organized, and saves time by eliminating manual downloads.

Related Articles

Data Analysis Page

GitHub Code

Kaggle Dataset Download Code