Database Design · School Assignment · University of Otago

MongoDB Scalable
Song Database

Designed and built a large-scale NoSQL database in MongoDB using a 498,000-track Spotify dataset — covering document schema design, data cleaning in Python, indexing strategy, and query design for a music player use case.

Back to Portfolio

At a Glance

498K Spotify tracks

1.3 GB Raw dataset size

4 Indexes designed

NoSQL MongoDB document model

The task was to design and implement a large-scale database system using a real-world dataset, justify the choice of database technology, and demonstrate practical database engineering skills — from raw data through to an indexed, queryable collection.

I selected a 498,000-track Spotify dataset from Kaggle, originally created to support a Retrieval-Augmented Generation (RAG) playlist system. Its semi-structured, nested format made it a good case study for evaluating NoSQL versus relational approaches.

Why MongoDB?

Before implementation, I evaluated three database paradigms against the dataset's structure and the target use case (a read-heavy music player application):

Approach	Fit for this project	Key reason
Relational (SQL)	Not ideal	Requires splitting into 5+ tables with joins — overly complex for simple filter queries
Graph database	Not ideal	Dataset is song-centric, not relationship-centric — no well-defined graph structure
MongoDB (document)	Chosen ✓	Flexible schema handles nested and optional fields; no joins needed; scales well for read-heavy use

MongoDB's PyMongo library also simplified the entire data cleaning and insertion pipeline, which was handled in Python before loading into the database.

Implementation Pipeline

Sample for design Extract ~5,000 records (1 per 500 entries) from the full dataset to validate schema design and cleaning logic before processing all 498K records.
Data cleaning in Python Parse raw JSON line by line. Split artist strings into lists, group audio features into nested documents, convert boolean contextual tags into arrays, and reformat similar-song references. Six fields were intentionally dropped (Emotion, Key, Tempo, Loudness, Time Signature, Explicit) to keep the schema simple for the target use case.
Schema design Design a flexible document structure — flat metadata at the top level, with Feature_Scores and Similar_Songs as nested sub-documents. No assumed completeness: missing fields are handled gracefully.
Database insertion via PyMongo Insert cleaned records into a MongoDB collection (songs in spotify_db). Drop and re-insert on each run to ensure consistency and avoid duplicates.
Indexing strategy Create single-field indexes on Artists, Genre, Album, and Contextual_Tags — targeting the most frequent query patterns while keeping write overhead low.
Query design and testing Implement and test representative music player queries — filter by artist, genre, or mood tag; aggregate audio features by genre; surface similar songs from embedded references.

MongoDB Python PyMongo JSON NoSQL Schema design Indexing

Challenges

Shell import was not repeatable — initial attempts using mongoimport via the MongoDB shell were difficult to customise and couldn't handle the cleaning logic needed. Switching to Python and PyMongo produced a modular, reusable pipeline that could be re-run cleanly.
Large dataset made early design difficult — working with 498K messy records from the start was impractical. Sampling one record every 500 entries gave a manageable 5,000-record subset for schema validation before applying the same logic to the full dataset.
Inconsistent artist name formatting — artist data mixed group names in quotation marks with individual names and unclear entries. Quotation marks were removed and treated as flat artist names; uncertain entries were retained to avoid premature data loss.

Outcome

The final database holds 498,000 Spotify tracks in a clean, indexed MongoDB collection — ready for typical music player queries without joins or schema migrations.

The Python-based pipeline is fully reusable: the same script can reload, re-clean, and re-insert the dataset with minimal modification if the data source changes.

The project demonstrated that technology selection should be driven by data structure and usage pattern — not by default preference for relational databases.