Music Exploration

Setup

Requirements

UV Tested with Python 3.11+

Config

Copy the config file example to instance folder

mkdir instance
cp config-example.py instance/config.py

Edit the config:

• Point the ROOT_DIR to the empty directory where all the data will be stored
• Point the AUDIO_DIR to the directory with all the audio files

If you are using own collection of audio, to serve audio from local server create a soft link app/static/audio pointing to your audio folder and make sure that AUDIO_PROVIDER is set to local.

ln -s /path/to/your/audio app/static/audio

If you want to visualize your local collection, you can run run.sh that will basically execute all the next steps automatically.

If you are using mtg-jamendo-dataset, you can serve audio directly from Jamendo servers by registering an app in Jamendo Dev portal and setting JAMENDO_CLIENT_ID variable. In this case make sure that AUDIO_PROVIDER is set to jamendo.

Environment

uv sync  # add --group processing and/or dev if you need

If you get an error while installing annoy, make sure that you have python3.x-dev installed

Additional processing libraries (processing dependency group):

• tensorflow-essentia: extracting embeddings (380MB):
• tinytag: install it if you use personal music collection, it parses ID3 tags

uv sync --group processing

Download essentia-tensorflow models

mkdir essentia-tf-models
cd essentia-tf-models
wget https://essentia.upf.edu/models/autotagging/msd/msd-musicnn-1.pb -O msd-musicnn.pb
wget https://essentia.upf.edu/models/autotagging/msd/msd-vgg-1.pb -O msd-vgg.pb
wget https://essentia.upf.edu/models/autotagging/mtt/mtt-musicnn-1.pb -O mtt-musicnn.pb
wget https://essentia.upf.edu/models/autotagging/mtt/mtt-vgg-1.pb -O mtt-vgg.pb
wget https://essentia.upf.edu/models/feature-extractors/vggish/audioset-vggish-3.pb -O audioset-vggish.pb

If you don't want to use all models, feel free to comment out entries in app/models.yaml

Embedding layers for the models:

Model	Layer	Size
MusiCNN	model/batch_normalization_10/batchnorm/add_1	200
VGG	model/flatten/Reshape	2 x 128 = 256
VGGish	model/vggish/fc2/BiasAdd	128

If you don't have a lot of tracks, feel free to add tsne to offline_projections in app/models.yaml. By default t-SNE is applied dynamically only on the number of tracks that you are visualizing at a time.

Process audio

To take full advantage of this application, make sure your audio is in the format that your browser can reproduce (mp3, ogg, etc.). If you need to batch convert your audio, check out this gist

uv run flask init-db  # creates tables in db
uv run flask index-all-audio  # creates list of audio tracks in db
uv run flask --group processing extract-all essentia-tf-models  # extracts embeddings
uv run flask reduce-all  # computes the projections
uv run flask index-all-embeddings  # indexes everything in database
uv run flask aggregate-all # aggregates embeddings in single .npy file per model (to get rid of many small files)

Adding local metadata

uv run flask --group processing load-id3-metadata

Adding Jamendo metadata

Clone or download metadata from mtg-jamendo-dataset

uv run flask load-jamendo-metadata path/to/mtg-jamendo-dataset/data/raw_30s_cleantags.tsv
uv run flask query-jamendo-metadata

Creating playlists

If you are using nix-based system, the playlist creation should work out of the box. If you want to create playlists for later use, change PLAYLIST_FOR_OFFLINE=True in config.py.

If you are using Windows Subsystem for Linux (WSL) and would like to use playlists in Windows, apart from setting PLAYLIST_FOR_OFFLINE=True, you should also set PLAYLIST_USE_WINDOWS_PATH=True and the path PLAYLIST_AUDIO_DIR='C:\\path\\to\\audio' that is the same one that you created symbolic link before, but as Windows path.

Note: none of these options have any effect if the AUDIO_PROVIDER='jamendo'.

Running the app

FLASK_ENV=development uv run flask run

Deploying with Docker (to be updated)

• Set SERVE_AUDIO_LOCALLY in config-docker.py appropriately depending on if you want to use Jamendo API or not
• Build and run docker image with your data mounted at /data, it uses port 80 by default
• If SERVE_AUDIO_LOCALLY=True, make sure to mount your audio dir at /app/static/audio otherwise make sure that you set JAMENDO_CLIENT_ID environment variable

Example:

docker build -t music-explore .
docker run -p 8080:80 -v /path/to/data:/data -v /path/to/audio:/app/static/audio music-explore  # run with local audio
docker run -p 8080:80 -v /path/to/data:/data --env JAMENDO_CLIENT_ID=XXXXXXXX music-explore  # run with Jamendo API

Or you can use the compose.yaml as a base.

Development

uv sync --dev
uv run prek install

Experiments

Some experiment commands are not runnable, particularly from hubness.py depend on scikit-hubness package, that is difficult to install.

License

The code is licensed under GNU Affero General Public License v3.0.

Citation

When using or referencing this work, please cite the following publication:

@inproceedings{tovstogan2020web,
  title = {Web Interface for Exploration of Latent and Tag Spaces in Music Auto-Tagging},
  author = {Philip Tovstogan and Xavier Serra and Dmitry Bogdanov},
  booktitle = {Machine Learning for Media Discovery Workshop (ML4MD), International Conference on Machine Learning (ICML)},
  year = {2020}
}

Acknowledgments

This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 765068.