Ontology-Guided Multimodal Framework for Explainable Music Similarity and Recommendation

Abstract

Analyzing music similarity in large catalogs is challenging because people perceive music differently and important details are found in audio, text, and metadata. This article introduces a multimodal framework that uses an ontology to make music similarity and recommendation more explainable. The framework brings together learned features from audio, lyrics, and other text with structured metadata in a shared similarity space, and then improves ranking with a music ontology that captures relationships between songs, artists, genres, and moods. The design works with any encoder that creates fixed-size features. This study uses strong neural audio and text encoders, mainly based on transformers. This approach allows the system to handle different input types while staying reliable across datasets. This study tests the framework on several open music and audio datasets using content-based retrieval tasks and standard ranking measures. In addition to Configurations C1–C4, this study includes an external content-based reference baseline based on conventional MIR audio descriptors. This baseline represents a signal-level retrieval approach that models complementary aspects of the audio signal, such as timbre, harmony, and spectral characteristics, and is evaluated under the same retrieval protocol as the main framework. It is included to provide an external comparison point outside the proposed C1–C4 design. Compared to audio-only and non-ontological variants within the same framework, the proposed multimodal and ontology-guided configurations achieve better precision, recall, and mean average precision, and also cover more rare content. Visualizations and case studies show that combining different data types and using ontology-based reranking can improve performance and make results easier to interpret. This work lays the groundwork for explainable, cognitively informed music recommendation systems and points to future work in modeling user behavior over time and adapting to different cultures.

1. Introduction

Music streaming services give listeners access to very large and diverse catalogues, making it easier to discover new artists, genres, and styles. Most music recommendation systems rely on collaborative filtering, content-based filtering, or hybrid combinations of the two. Despite their practical success, several persistent problems remain. Recent surveys of music recommender systems [1,2] highlight recurring challenges such as cold start, data sparsity, and the difficulty of maintaining robust recommendation quality in dynamic real-world environments. These limitations make it harder for recommendation systems to deliver consistently relevant results and, in turn, can reduce user satisfaction.

A further difficulty is that music recommendation must balance several objectives at once. Accuracy alone is not enough: systems are also expected to support diversity, novelty, contextual relevance, and some degree of transparency. This becomes harder when recommendations are based on heterogeneous inputs such as audio, text, metadata, user history, and contextual signals. Recent surveys show a clear shift toward multimodal recommendation, including the use of audio signals, textual metadata, images, and contextual features [1,2,3]. At the same time, studies on popularity and exposure bias, on the cultural effects of recommendation algorithms, and on explainability in recommender systems show that optimising only for accuracy can lead to narrow listening habits, limited exposure to diverse or less popular music, and opaque recommendation behaviour from the user’s perspective. These tensions motivate recommendation frameworks that are not only effective, but also more interpretable and controllable.

Music similarity is a central part of this problem. Similarity models help determine which tracks, artists, or genres are likely to be perceived as related, and they often serve as the basis for retrieval, recommendation, and discovery. Recent neural models, including transformer-based encoders and other high-capacity architectures, have substantially improved audio and multimodal representation learning. However, these advances do not automatically make music recommendations easier to interpret. Many existing systems still combine modalities in relatively shallow ways, use metadata only as auxiliary features, or rely on latent representations that provide little explanation of why two items are treated as similar. As a result, strong performance and clear semantic interpretation often remain separate goals.

This article proposes an ontology-guided multimodal framework for explainable music similarity and recommendation. The framework combines audio content, textual descriptions, and structured metadata in a shared similarity pipeline and supports different encoder backbones, including transformer-based models and a lightweight convolutional baseline. On top of the learned multimodal representation, it introduces a lightweight ontology-guided semantic layer for interpretable reranking. In the present implementation, this semantic layer is intentionally narrow: it focuses on a small set of music-relevant entities and relations and is used to provide explicit semantic control and explanation rather than deep formal ontology reasoning.

The novelty of the article lies primarily in its integration and evaluation framing rather than in ontology engineering alone. More specifically, the study combines multimodal music representations and a lightweight ontology-guided semantic layer within a single explainable retrieval framework; evaluates this framework across several public datasets with different semantic and annotation characteristics; and analyses when multimodal fusion and ontology-guided reranking improve retrieval quality, diversity, and interpretability, and when their effect remains limited. In this sense, the contribution is not a new formal music ontology in itself, but an empirically grounded framework for linking modern multimodal representation learning with interpretable semantic control in music similarity and recommendation.

Some of the ideas explored here were introduced in an earlier conference paper by the author [4]. The present article substantially extends that work. It provides a fuller encoder-agnostic description of the architecture, clarifies the role and limits of the ontology-guided semantic layer, and reports a broader empirical study with additional internal comparison settings, modality analyses, and qualitative inspection of the learned embedding space. The overall goal is to make music similarity modeling more transparent, more controllable, and easier to analyse across different dataset conditions.

The remainder of the paper is organized as follows. Section 2 reviews related work on music recommendation and similarity modeling. Section 3 describes the proposed framework and methodology. Section 4 presents the experimental setup and results. Section 5 discusses the findings, limitations, and directions for future work.

2. Related Work

Research on music recommendation has evolved along several lines, reflecting the increasing complexity of musical content, user preferences, and real-world application scenarios. This section reviews the main approaches presented in prior work, with an emphasis on the limitations that motivate the framework proposed in this study.

2.1. Collaborative Filtering, Hybrid Approaches, and System-Level Biases

Collaborative filtering and hybrid recommendation systems form the basis of most large-scale commercial platforms. They model user–item interactions effectively when historical data is abundant, but they also have well-known weaknesses. Surveys of music recommendation [1,2] and content-driven approaches [5] show that these systems struggle in cold-start scenarios, where users or items lack interaction history, and that their reliance on collective behavior can worsen popularity bias. Recent studies and surveys on popularity bias and long-tail effects in music recommendation [6,7] show that these models often overrepresent mainstream content and underexpose items in the long-tail, including many lesser-known artists. This limits diversity and cultural representation in recommendation lists.

Social and behavioral factors further complicate recommendation quality. Recent surveys on the user cold-start problem in recommender systems [8] show that even advanced meta learning, cross-domain, and hybrid strategies only partially address this challenge, especially when privacy constraints limit the use of rich side information. At the same time, fairness-oriented analyses of music recommender systems [9] and broader cultural and policy studies [10,11] show that algorithmically driven recommendations can reinforce existing hierarchies in the music ecosystem, affect the visibility and earnings of different groups of artists, and shape listener behavior in opaque ways. These findings motivate methods that not only optimise accuracy, but also provide greater fairness, interpretability, and user-level control over recommendation outcomes.

2.2. Content-Based and Music Information Retrieval Approaches

Content-based methods model tracks by analyzing audio directly. Early research in Music Information Retrieval (MIR) laid the groundwork for describing music using low- and mid-level signal features. Surveys by Casey et al. [12] and Schedl et al. [13] cover content-based MIR techniques for timbre, pitch, rhythm, and structure, as well as their applications in similarity search and browsing. Newer toolkits like MIRFLEX [14] bring together advanced feature extractors in one library, which helps researchers combine signal-level descriptors, high-level musical features, and learned representations for large-scale studies.

Building on these advances, deep learning has led to better audio representations. Costa et al. [15] found that convolutional neural networks working with spectrograms can outperform traditional hand-crafted features in music classification. Hershey et al. [16] compared different CNN architectures on large sound datasets and developed strong models for general audio embedding. Cramer et al. [17] explored how to design self-supervised deep audio embeddings and showed that these models can be used for many tasks, even with limited labeled data. While these studies offer powerful content encoders, they are usually designed for general audio or single-task classification, and do not include ontologies or user-focused explanation tools for music recommendation.

2.3. Hybrid, Graph-Based, and Deep Learning Recommender Systems

Hybrid recommenders use both collaborative filtering and content-based signals, combining interaction data with item details. Recent graph-based hybrid models represent users, items, and side information in a single graph and use graph neural networks (GNNs) to share preferences across complex connections. In music, Bevec et al. [18] found that a GNN-based hybrid model outperforms traditional collaborative and content-based methods for next-track prediction. Recent work on explainable graph-based recommender systems [19] further shows how graph structure can support more transparent recommendation logic and more interpretable relations among users, items, and attributes.

Deep learning and structured semantic modeling have broadened how recommender systems are designed. Zhang et al. [20] review recommender systems based on knowledge graph embedding and show how graph-based representation learning can improve recommendations by encoding structured relations among users, items, and attributes. Uta et al. [21] provide a recent overview of knowledge-based recommender systems and highlight their importance for explainability, constraint-aware recommendation, and cold-start settings. Together, these studies indicate that recent recommender research increasingly combines learned representations with explicit semantic structure. However, such systems still rarely frame multimodal information as part of a unified and interpretable similarity space, which remains particularly important for music recommendation and similarity analysis.

2.4. Emotion- and Context-Aware Music Recommendation

Another line of research focuses on adapting recommendations to a user’s emotional state or situational context. EEG-based brain–computer interface (BCI) studies, such as the survey by Gu et al. [22], show that physiological signals can help infer cognitive and emotional states that may guide media choices. Some music recommender systems now model user emotion and music mood more directly. Recent developments include systems that use live facial images to detect emotion and adapt recommendations accordingly [23], as well as deep probabilistic models that account for differences in users’ emotional states and mood preferences [24].

Context-aware music recommendation considers factors beyond emotion, including location, device, and cultural background. Schedl et al. [25] use country-level listener types and show that incorporating such context can affect both recommendation accuracy and the kinds of music that are recommended. A broader review of context-aware recommender systems [26] highlights continuing challenges in representing context, combining it effectively with user data, and evaluating its impact on user experience.

These emotion- and context-aware approaches illustrate the promise of more fine-grained user modeling, but they also face practical limitations. They may require specialised sensors or potentially intrusive data collection, remain vulnerable to errors in emotion inference, and often operate in ways that are difficult for users to interpret. The approach proposed in this article takes a different path by relying on semantic and ontological structure to make similarity modeling and recommendation more transparent and controllable, without depending on physiological signals or detailed personal sensor data.

2.5. Multimodal and Transformer-Based Representation Learning for Music

Recent advances in transformer architectures and large-scale multimodal learning have had a strong impact on music retrieval and recommendation. Contrastive Language–Audio Pretraining (CLAP) [27] learns joint representations of audio and natural language descriptions by training separate audio and text encoders with a contrastive objective on large collections of sound–caption pairs. This yields a general-purpose audio–text embedding space that supports zero-shot classification and retrieval across music, environmental sounds, and speech. More recent work such as TALKPLAY [28] extends this direction toward multimodal music recommendation by integrating audio, lyrics, metadata, semantic tags, and playlist co-occurrence within a large language model framework. More specialised music encoders, including MERT [29], use large-scale self-supervised pretraining with teacher–student objectives to learn acoustic representations that outperform conventional speech- or audio-based baselines on a wide range of music understanding tasks.

Cross-modal alignment for music has also been explored in models such as MuLan [30], which maps music audio and natural language into a shared embedding space optimised for tagging and retrieval. Trained on large-scale music–text pairs, MuLan supports zero-shot tagging and text-to-music retrieval while remaining compatible with conventional tag vocabularies. More recent work continues this trend toward richer multimodal music modeling. SUMMR [31], for example, proposes a unified multimodal representation framework for songs based on shared-space embedding and cross-modal attention. These developments further support the relevance of multimodal representation learning for music recommendation, while also underscoring the need for methods that connect such learned spaces to explicit semantic and ontological reasoning.

Taken together, these models demonstrate that high-capacity transformer and multimodal architectures can provide strong audio and music representations and flexible cross-modal interfaces. However, they are typically designed as general-purpose encoders or representation models rather than as complete, explainable recommendation frameworks. They do not, by themselves, include ontology-based reasoning or explicit mechanisms for structuring and explaining similarity relationships between songs, artists, genres, and moods—gaps that the framework proposed in this article aims to address.

2.6. Summary and Research Gaps

Although there has been substantial progress in collaborative, content-based, graph-based, emotion-aware, and multimodal recommendation approaches [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31], several important challenges remain. Most multimodal and transformer-based models focus on learning representations and aligning heterogeneous data sources, but they rarely explain why particular items are considered similar or recommended. Methods based on contrastive learning, late fusion, or large pretrained encoders also make it difficult to incorporate structured domain knowledge, such as musical ontologies or taxonomies, in a transparent and controllable way. More broadly, relatively few systems address popularity and exposure bias, support culturally sensitive recommendations, or provide user-facing explanations that make similarity judgments easier to interpret.

These gaps help explain the methodological choices made in this study.

Table 1. Limitations in Existing Approaches and Objectives of the Proposed Framework.

Limitation in Prior Work	Description	Objective Addressed in This Study
Cold-start and sparsity	Recommendation quality degrades when there is little or no interaction history for users or items.	Rely on multimodal content representations (audio, text, metadata) so that similarity can be estimated even without extensive user history.
Popularity bias	Models tend to overexpose highly popular items and underrepresent long tail content, reducing diversity and fairness.	Use ontology-guided similarity and diversity-aware ranking to prioritize semantically relevant items rather than purely popularity-driven choices.
Shallow semantic modeling	Many systems focus on low-level acoustic cues or coarse labels that only weakly reflect perceived musical similarity and intent.	Employ richer semantic representations aligned with a music ontology to better capture high-level concepts and listener perceptions.
Limited interpretability	Deep and graph-based models typically operate as black boxes and provide little insight into why particular items are recommended.	Add an ontology-based reasoning layer that maps latent representations to human-understandable musical concepts and supports explanatory feedback.
Insufficient cultural and contextual adaptability	Recommendation algorithms often favor dominant genres, regions, and usage scenarios, overlooking cultural and contextual diversity.	Use structured domain knowledge and multimodal representations to support culturally aware similarity neighborhoods and configurable diversity criteria.

summarizes the main limitations identified in prior work, including cold-start problems, popularity bias, limited explainability, and weak integration of structured semantic knowledge. It also shows how these limitations motivate the design goals of the proposed ontology-enhanced multimodal framework.

The framework introduced in this study addresses these challenges by combining audio, text, and metadata with an ontology-guided reasoning layer that links learned embeddings to a more explicit and interpretable similarity structure. In this sense, the proposed approach builds on earlier work in music recommendation and multimodal representation learning [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31] while aiming to provide a more transparent, semantically grounded, and controllable basis for music similarity analysis and recommendation.

3. Materials and Methods

This section introduces the proposed ontology-enhanced multimodal framework for music similarity and recommendation. It describes the architecture, input types, neural encoders, multimodal fusion, ontology-based semantic layer, similarity calculation and ranking, as well as implementation details.

The study focuses on three main research questions. First, it looks at whether combining audio, text, and structured metadata improves music similarity and retrieval compared to using only audio. Second, it tests if an ontology-based similarity layer can boost diversity and help users discover less popular items, while still keeping recommendations relevant compared to embedding-based ranking. Third, it checks if the framework can work well with different encoder backbones, from small convolutional networks to large pre-trained audio and text models, without changing the overall design.

To answer these questions, the study compares different versions of the framework that gradually add multimodal fusion and ontology-aware re-ranking (see Section 3.7). These versions are tested on public music and audio datasets (see Section 4). The following methods are described in enough detail to allow others to reproduce and build on this work.

3.1. Overall Framework

The framework uses a modular pipeline that can work with different audio and text encoders, while keeping a consistent similarity and reasoning layer. The system has four main parts. First, an input processing stage prepares audio segments, text descriptions like lyrics and tags, and structured metadata for encoding, following standard practice in music information retrieval and multimodal recommendation systems [3,12,13]. Second, neural encoders for each type of input map them to a fixed-size embedding space using pre-trained or lightly fine-tuned models, as seen in recent audio and audio–text representation learning research [15,16,17,27,28,29,30,31]. Third, a multimodal fusion block brings these embeddings together into a shared similarity space. Finally, an ontology-based similarity layer matches the fused representations with a music ontology, using this structure to improve similarity scores and provide explainable recommendations, building on the earlier conference version of the framework [4].

Figure 1 shows a diagram of the system’s architecture. At the top, audio, text, and metadata inputs go through their own encoders to create specific embeddings for each type. These embeddings are merged in a multimodal fusion block, resulting in a single combined representation in a shared similarity space. This space is used for nearest-neighbour retrieval and ranking. On top of this, the ontology layer connects song nodes to artists, genres, and moods using links like hasArtist, hasGenre, and hasMood. These connections help calculate an ontology-based compatibility score Φonto(q,i) (explained in Section 3.6). This score can rerank candidate tracks and provide simple, concept-level reasons for recommendations.

3.2. Input Modalities and Data Representation

The framework supports three main groups of input signals: audio content, textual descriptions, and structured metadata.

Audio signals are represented as raw waveforms or time–frequency transforms, such as log-mel spectrograms, depending on the encoder. For retrieval experiments, fixed-length excerpts of 10 to 30 s are extracted from each track, with optional segment overlap. All audio is downsampled to a common rate, such as 22.05 kHz, to balance fidelity and computational cost across datasets and models.

Textual information includes lyrics when available, as well as short descriptions, tags, or other textual metadata associated with a track. Text is normalised (lowercasing and basic cleaning), tokenized, and truncated or padded to a maximum length compatible with the selected text encoder. This yields a sequence of tokens that can be converted into a dense representation by a sentence-level transformer model, as described in Section 3.3.

Structured metadata covers both categorical and numeric fields, such as artist identifiers, genre labels, release year, tempo, or high-level mood tags. Categorical variables are encoded as learned embeddings or one-hot vectors, while numeric variables are normalized to zero mean and unit variance before being passed through a small multilayer perceptron. When a dataset provides an explicit symbolic genre or mood taxonomy, these labels are later aligned with the ontology described in Section 3.5, so that the same information can be used both as input features and as semantic constraints during re-ranking.

In the experiments, these inputs are derived from public music and audio datasets described in Section 4. The methods are not tied to a specific corpus and can, in principle, be applied to other collections with a similar combination of audio, text, and metadata.

3.3. Neural Encoders

The framework is designed to be encoder-agnostic, meaning any model that maps an input modality to a fixed-size vector can serve as its backbone. This study analyzes a combination of neural audio and text encoders that follow current best practices in music information retrieval and multimodal representation learning [12,13,27,28,29,30,31], along with a simpler convolutional baseline. This setup makes it possible to distinguish the impact of the encoder choice from the effects of multimodal fusion and ontology-based reasoning.

3.3.1. Audio Encoders

For audio, two main encoder families are considered. First, a lightweight convolutional spectrogram encoder is used as a baseline. Audio signals are converted to log-mel spectrograms and passed through a compact stack of convolutional and pooling layers, followed by global pooling to obtain a fixed-dimensional embedding. This configuration follows common designs in CNN-based music and audio classification [15,16,17], which have been shown to provide strong and relatively lightweight content representations. A final linear layer projects the resulting vectors to the shared dimensionality d, so that they can be combined with other modalities in the fusion module.

Second, the reported experiments use a pretrained audio–text encoder that provides richer semantically informed audio representations. In the main experimental setup, the audio backbone is the published CLAP model laion/clap-htsat-unfused [27], initialized from a publicly available checkpoint. CLAP learns a joint embedding space for audio and natural language descriptions and has been shown to support flexible cross-modal retrieval and zero-shot audio understanding [27]. In the present framework, only the audio branch of the pretrained model is used at retrieval time, and its output is projected to the shared d-dimensional space by a shallow linear layer.

The framework is architecturally compatible with other pretrained music encoders, including music-specific models such as MERT [29] and joint music–text encoders such as MuLan [30], provided that their outputs are projected to the same shared space. However, unless otherwise stated, these models are discussed as compatible alternatives rather than as part of the main reported experimental configuration.

In all reported experiments, pretrained encoder parameters are initialized from publicly available checkpoints and kept frozen. Only shallow projection layers to the shared space are trained. This stabilizes optimization on relatively modest music datasets and focuses the evaluation on the effects of multimodal fusion and ontology-based reranking rather than on the details of large-scale pretraining.

3.3.2. Text Encoders

Textual descriptions (lyrics, tags, and short free-form descriptions) are encoded using a sentence-level transformer language model. The framework is compatible with other BERT-family and SBERT-style encoders [32,33]. In the reported experiments, the default text encoder is sentence-transformers/all-MiniLM-L6-v2 [33], which produces dense vector representations suitable for semantic similarity and retrieval.

For short tags and descriptions, each text item is tokenised and fed to the encoder; a single vector is then derived from the [CLS] token or by mean-pooling over token embeddings. For longer lyrics, the same procedure is applied, with truncation to a maximum sequence length compatible with the encoder. The resulting text embeddings are projected, if necessary, to the shared dimensionality d to match the audio and metadata representations. Because the framework treats the text encoder as a black box that outputs fixed-size vectors, alternative backbones (for example, music-adapted or multilingual models) can be substituted without modifying the rest of the architecture.

3.3.3. Metadata Representation

Structured metadata is represented using a combination of learned embeddings and normalised continuous features. Categorical attributes such as artist identifiers, genre labels, and discrete mood tags are mapped to low-dimensional embedding vectors and concatenated. Numeric attributes such as tempo, track duration, or release year are first normalised to zero mean and unit variance and then passed through a small multilayer perceptron to obtain a compact continuous representation.

The resulting metadata vector can be treated as an additional modality in the fusion module, on the same footing as audio and text. When metadata fields correspond directly to symbolic categories included in the ontology (for example, genre or mood labels), they are also used to instantiate nodes and relations in the semantic layer described in Section 3.5. This dual role allows the same information to contribute both to the continuous similarity space and to the ontology-based compatibility score used for re-ranking.

3.4. Multimodal Fusion

Multimodal fusion aims to combine embeddings from audio, text, and metadata into one representation that brings together information from each type. Here, a, t, and m stand for the projected embeddings for audio, text, and metadata, each in a d-dimensional space.

The main setup uses a straightforward concatenation and projection method. The vectors for each modality, (a; t; m), are joined together and sent through a linear layer, sometimes followed by a non-linear function, to return them to dimension d. This keeps information from each modality and lets the model learn how much each one should influence the final result.

To allow the contribution of each modality to adapt to the input, this study also considers gated or attention-based fusion. In this case, a small network computes non-negative modality weights that sum to one, and the fused vector is a weighted sum of the modality embeddings. This follows the general principle of feature-level attention for multimodal fusion, where attention scores indicate which modalities are most informative for a given instance [34]. Such mechanisms are particularly useful when lyrics or textual descriptions are absent, or when metadata fields are sparse or unreliable.

When using pretrained audio–text encoders such as CLAP [27] or related joint music–text models [30], the audio and text embeddings are already partly aligned in a shared space. For these models, fusion can use a simpler projection layer, and more focus can go to matching metadata features and ontology concepts with the audio–text space. In the experiments in Section 4, concatenation with projection is the main fusion method because it is simple, stable, and clear. Attention-based fusion is also tested in some cases to see if dynamic weighting of modalities offers any benefits.

3.5. Music Ontology and Semantic Layer

The ontology-based semantic layer has two main roles. It organizes the music domain by defining entities such as songs, artists, genres, and moods, together with their relationships. It also connects the continuous embedding space learned by the encoders to human-interpretable concepts, making similarity judgments and recommendations easier to explain.

At the practical level, the semantic layer provides a concept-level decomposition of similarity, for example, through artist, genre, and mood agreement. This information can be presented alongside the embedding-based score, making the hybrid ranking method easier to interpret, inspect, and control.

The ontology used in this study is deliberately lightweight. It focuses on a small core of music-relevant entity types (Song, Artist, Genre, and Mood) and a corresponding set of semantic relations that can be aligned reliably with public dataset metadata. In its current form, the semantic layer should be understood as an ontology-guided representation of structured musical knowledge rather than as a fully developed formal ontology with rich logical axioms, deep genre hierarchies, or extensive subgenre reasoning.

Ontologies and controlled vocabularies for music and sound semantics have been studied extensively, ranging from early domain-specific music ontologies for mood and situation reasoning in recommendation systems [35] to more recent surveys of sound and music taxonomies and ontological resources. The present work adopts a lightweight ontology that focuses on a core set of classes (Song, Artist, Genre, Mood) and relations (hasArtist, hasGenre, hasMood, similarTo). Existing metadata fields in the datasets are mapped to these classes and relations, and additional semantic links can be added manually or mined from external resources when available.

Dataset-to-ontology mapping. In the experiments, each track is represented as a Song node. The metadata available in a dataset is then mapped to ontology entities in a straightforward way: artist IDs create Artist nodes connected via hasArtist, genre labels create Genre nodes connected via hasGenre, and mood tags (when provided) create Mood nodes connected via hasMood. If some attributes are missing (for example, a dataset only has genre), the ontology score is computed only from the available information rather than treating missing fields as negative evidence. This keeps the semantic layer usable even when datasets differ in annotation richness.

Each song in the ontology is linked to its combined embedding. Artist and genre nodes are either represented by the average of their songs’ embeddings or treated as symbolic nodes that exist only in the graph. Similarity scores between songs are then adjusted using the ontology. For instance, if two tracks are close in the embedding space and share the same genre and mood in the ontology, their similarity score increases; if they are close in the embedding space but differ strongly in ontology attributes, the score decreases.

Unlike earlier ontology-driven recommenders, which often hard-code rule-based reasoning over symbolic descriptors [35], the proposed layer is designed as a soft reranking mechanism that modulates rather than replaces embedding-based similarity. Compared with established music ontologies and broader sound-semantic taxonomies in the literature [35], the present approach adopts a narrower and more operational semantic scope designed primarily for interpretable reranking rather than for formal knowledge representation in its own right. This design keeps the system compatible with high-capacity neural encoders while still allowing recommendations to be explained in terms of familiar concepts such as artist, genre, and mood. In practice, the ontology layer enables operations such as ontology-aware reranking of nearest neighbours, grouping and filtering recommendations by semantic attributes, and generating simple explanations of the form: recommended because it is acoustically similar and shares genre X and mood Y with the seed track.

In particular, the implementation used in the present experiments does not yet model hierarchical genre taxonomies, graded semantic similarity between related subgenres, or ontology reasoning in a formal language such as OWL or RDF/description logic. Instead, the semantic contribution is operationalised through a small set of interpretable symbolic relations and attribute-level compatibility signals. This design choice favors robustness and reproducibility across heterogeneous public datasets, but it also limits the expressive richness of the ontology layer.

3.6. Similarity Computation and Ranking

For each track i, the framework produces a fused embedding vector $z_i\in {\mathbb{R}}^d$ in the shared similarity space. At query time, given a seed track q with embedding $z_q$ candidate tracks are first ranked by cosine similarity in this space, following current practice in vector retrieval systems [36]. Cosine similarity between two embeddings $z_q$ and $z_i$ is computed as

$$cos(z_q,z_i)={\displaystyle \frac{z_q\cdot z_i}{\parallel z_q{\parallel }_2 \parallel z_i{\parallel }_2}}$$

(1)

Here “·” denotes the dot product between vectors, and $\parallel \cdot {\parallel }_2$ denotes the Euclidean norm. Because all embeddings are normalized to unit length, cosine similarity effectively depends on the angular relationship between the two vectors and is widely used in modern vector retrieval settings [36].

Nearest-neighbor search over the embedding index is implemented with an approximate nearest-neighbour backend suitable for high-dimensional vectors, following current practice in ANN indexing systems [37]. In the experiments, the top-k nearest neighbors of each query are retrieved according to Equation (1), and an ontology-aware re-ranking step is then applied.

To combine pure embedding similarity with semantic constraints from the music ontology, the final score $S(q,i)$ for a candidate track i is defined as

$$S\left(q,i\right)=\lambda cos\left(z_q, z_i\right)+\left(1-\lambda \right) {\Phi }_{onto\left(q,i\right)}$$

(2)

where λ ∈ [0, 1] controls the balance between the two terms and ${\Phi }_{\mathrm{onto}}(q,i)$ is an ontology-based compatibility function. The value of λ = 0 corresponds to purely embedding-based ranking, while λ = 1 uses only ontology-based similarity. Intermediate values allow the model to trade off between what the neural encoders consider similar and what the ontology considers semantically consistent.

The ontology component ${\Phi }_{\mathrm{onto}}(q,i)$ aggregates agreement on selected semantic attributes:

$${\Phi }_{onto\left(q,i\right)}= w_{artist}{I}_{artist\left(q,i\right)}+w_{genre}{I}_{genre\left(q,i\right)}+w_{mood}{I}_{mood\left(q,i\right)}$$

(3)

Here $I_{\mathrm{artist}}(q,i)$, $I_{\mathrm{genre}}(q,i)$, and $I_{\mathrm{mood}}(q,i)$ are indicator functions (or graded similarity scores) that quantify whether the query and candidate tracks share the same artist, genre, or mood in the ontology, respectively. In the simplest case, these indicators take the value 1 when the attribute matches and 0 otherwise. The coefficients $w_{\mathrm{artist}},w_{\mathrm{genre}},w_{\mathrm{mood}}\ge 0$ are non-negative weights that determine the relative importance of each semantic dimension. These weights, together with the balance parameter λ, are tuned on a validation set.

This formulation makes it possible to analyze how much ontological information contributes to retrieval quality and to control the trade-off between relevance, diversity, and exposure of less popular items. The ranked list used for evaluation is obtained by sorting candidates in descending order of the final score.

Section 3.7 specifies how this scoring function is embedded in the training and evaluation protocol, and Section 4 reports its empirical impact relative to purely embedding-based baselines and other neural ranking variants [38].

3.7. Training and Evaluation Protocol

The framework can be used with frozen pre-trained encoders and a learned fusion layer, or by jointly fine-tuning selected parts. In this study, the main goal is to analyze the framework, not to focus on large-scale pre-training. Audio and text encoders are set up using publicly available pre-trained checkpoints and are either kept frozen or lightly fine-tuned for music retrieval tasks, depending on the experiment. In every setup, the fusion layer, metadata projection network, and ontology-based re-ranking parameters are trained on content-based retrieval tasks from public datasets described in Section 4. The balance parameter λ in Equation (2) and the ontology weights $w_{\mathrm{artist}},w_{\mathrm{genre}},w_{\mathrm{mood}}$ in Equation (3) are tuned on a validation set by grid search over a small range of candidate values.

The training process uses a contrastive objective, which helps similar items get embeddings that are close together and pushes dissimilar items further apart. This approach is common in representation learning for retrieval and recommendation tasks. In each mini-batch of tracks, positive pairs come from items that are similar in the dataset, such as tracks from the same album, playlist, or artist. Negative pairs are chosen from other items in the batch. The model uses a version of InfoNCE or triplet loss to increase the cosine similarity in Equation (1) for positive pairs compared to negatives. This contrastive training method has proven effective for learning embeddings that work well for nearest-neighbour retrieval in many areas, including multimodal and recommender systems [39,40].

To understand how each part of the framework contributes, four configuration setups are tested. In the first configuration (C1), only the convolutional spectrogram encoder is used. Tracks are ranked by the cosine similarity of their audio embeddings, and text, metadata and ontology-based re-ranking are disabled. In the second configuration (C2), the CNN is replaced with a pre-trained audio or audio–text encoder projected into the shared space, but other modalities and the ontology are still not used. The third setup (C3) used the full multimodal fusion module, combining audio, text, and metadata embeddings into one representation, while ranking was still based only on embeddings as described in Equation (1). The fourth configuration (C4) corresponds to the complete framework proposed in this article: multimodal fusion is followed by ontology-based re-ranking using the score $S(q,i)$ defined in Equation (2). The comparison between C1–C4 thus isolates the effect of stronger audio encoders, multimodal fusion, and ontology-aware reasoning, respectively.

Evaluation follows standard practice in recommender systems and information retrieval. For each query track in the test set, the system retrieves a ranked list of candidate tracks, and computes precision, recall, and F1-score at a cutoff k (for example, k = 10 or k = 50), together with mean average precision at cutoff k (MAP@k). These metrics are used to assess the accuracy and ranking quality of recommender systems [41]. Where appropriate, results are additionally stratified by popularity or genre to analyze how different configurations affect exposure to long-tail content and the balance between mainstream and niche items.

To assess variability across queries, ranking metrics are computed over all query tracks in each test split. In the main aggregated results, 95% bootstrap confidence intervals based on 1000 resamples are reported for the principal retrieval metrics, as described in Section 4. These intervals support a more cautious interpretation of configuration differences, especially where gains are small. This approach helps relate observed retrieval performance to variability in the query set while keeping the training setup unchanged.

Train, validation, and test splits are constructed at the artist or album level, depending on the dataset, to reduce leakage between sets and to assess generalization to genuinely unseen material. All hyperparameters, training schedules, and early stopping criteria are kept identical across configurations C1–C4 unless explicitly stated, so that observed performance differences can be attributed to the choice of encoder, multimodal fusion, and ontology-based re-ranking rather than optimization artefacts.

3.8. Implementation Details and Reproducibility

All experiments are implemented in Python (version 3.10) using the PyTorch (version 2.1) deep learning framework, which provides automatic differentiation and GPU acceleration for tensor-based models [42]. Training and inference are executed on GPU-enabled machines, with batch sizes and learning rates selected through preliminary validation experiments under memory constraints. For the convolutional baseline, a compact architecture with a small number of convolutional blocks is used in order to keep the model lightweight and easy to reproduce. For the pre-trained audio and audio–text encoders, only shallow projection and fusion layers are trained, while the backbone parameters remain frozen. This design stabilizes optimization and reduces the risk of overfitting on the moderately sized music datasets used in this study.

The implementation follows common recommendations for reproducible computational research [43]. Random seeds are fixed for all stochastic components where possible, and model configurations are defined via explicit configuration files that specify the encoder type, fusion strategy, ontology parameters, and training hyperparameters. Training logs and validation metrics are stored for each run, together with information about the software environment (Python and library versions) and hardware used. Approximate nearest-neighbor search backends and evaluation scripts are integrated into the same code base so that the full retrieval pipeline—from raw inputs to ranked lists and metrics—can be executed with a single configuration.

No new proprietary datasets are introduced. All datasets used in the experiments are public and are described in Section 4, together with licensing and access information. The pretrained encoders used in the reported experiments are obtained from publicly available model repositories. In particular, the audio backbone is the published CLAP checkpoint laion/clap-htsat-unfused, and the default text encoder is sentence-transformers/all-MiniLM-L6-v2. The source code and experiment configuration files are maintained in a version-controlled repository. An anonymized version of the repository will be made available during peer review, and, upon acceptance, the full code and configuration files will be released under an open-source license and archived with a persistent identifier to facilitate long-term access. Additional implementation details, including pre-processing settings, encoder configurations and training schedules, are provided in Appendix A.

In line with the journal’s guidelines on the use of generative artificial intelligence, generative AI tools were not used to create or modify datasets, to train models, or to compute evaluation metrics. Language models were employed only for minor text editing, such as improving grammar and clarity, and did not influence the experimental design, the implementation of the framework, or the interpretation of the results.

The next section describes the datasets, experimental setup, and evaluation scenarios used to assess the proposed framework.

4. Results

This section explains how the experiments for the ontology-enhanced multimodal framework are set up and what can be observed from them, both quantitatively and qualitatively. The analysis focuses on the three questions introduced in Section 3: whether combining several content modalities improves content-based similarity over audio-only baselines; how much the ontology-based similarity layer helps with diversity and long-tail exposure; and how robust the framework is when the underlying encoder backbones are changed.

All experiments are run in a single PyTorch (version 2.1) codebase that reuses the same preprocessing, batching, and optimization routines for every model. This keeps implementation differences to a minimum and makes the comparisons between configurations fairer and more trustworthy.

4.1. Datasets

This study uses four public datasets, each highlighting a different aspect of music similarity and recommendation: large-scale metadata and popularity, genre classification, handling background sounds, and realistic long-tail music catalogues.

The Million Song Dataset (MSD) [44] offers metadata and precomputed audio features for one million commercial tracks. Since it does not provide raw audio in the same form as the other datasets, MSD is not used to train waveform-based audio encoders. However, it is included in the retrieval analysis through its available precomputed audio features together with metadata, which makes it useful for large-scale statistical analysis, long-tail evaluation, and comparative retrieval analysis within the shared framework.

GTZAN [45] is a well-known benchmark for genre classification, with 1000 audio clips, each 30 s long, spread evenly across ten genres. Although it has some duplicated tracks and label noise, it remains useful for controlled tests of genre similarity. In this work, GTZAN helps check if the learned embeddings group by genre and how genre labels work with the ontology layer.

FSD50K [46] is an open dataset of human-labeled sound events with a wide license. While it covers more than just music, it includes many musical and similar sounds. These clips are used to test how well the audio encoder and multimodal framework handle situations where music and non-musical sounds mix, similar to real-world listening.

The Free Music Archive (FMA) [47] is a large set of Creative Commons-licensed music with detailed metadata. This study uses the small and medium FMA subsets as the main sources of raw audio, genre and artist labels, descriptions, and other metadata. FMA is the main benchmark for training and testing the hybrid model in realistic settings, including a long-tail artist distribution and varied genres. FMA is used as the primary benchmark for audio-based baselines requiring raw waveform access.

All datasets are used according to their licenses and terms. No personal user logs or identifiable listener data are included; only the content and metadata from the datasets are processed.

Table 2. Overview of datasets used in the experiments.

Dataset	Domain	Size	Modalities Used	Role in Experiments
MSD [44]	Commercial music catalogue	1 M tracks	Metadata, precomputed audio features	Source of metadata and precomputed audio features; used in retrieval analysis, long-tail evaluation, and metadata-based modeling
GTZAN [45]	Genre-labelled music	1000 clips, 30 s	Audio, genre labels	Controlled analysis of genre-based similarity and embedding structure
FSD50K [46]	General sound events (including music)	51,000 clips approximately	Audio, event labels	Robustness of audio encoders and multimodal models to non-musical and mixed content
FMA [47]	Creative Commons music	Small/medium subsets, thousands of tracks	Audio, artist, genre, year, textual metadata	Main benchmark for training and evaluating multimodal similarity and ontology-aware re-ranking

summarizes the main characteristics and roles of these datasets in the experiments.

4.2. Data Preparation

Data preparation follows a unified procedure across datasets to ensure that differences in performance can be attributed to the models rather than to preprocessing choices.

For audio-based experiments, each track is segmented into fixed-length excerpts. Pilot experiments indicated that very short windows (1–5 s) fail to capture higher-level structure, whereas very long excerpts considerably increase memory requirements and training time. In the reported experiments, excerpts of 10–30 s are used, depending on the dataset and task. All audio is resampled to 22.05 kHz, which offers a good trade-off between fidelity and computational cost for music signals. For spectrogram-based encoders, log-mel spectrograms are computed with a 1024-sample window, 75% overlap, and 64 mel bands. Segment boundaries are aligned so that each excerpt has a consistent representation across audio, text, and metadata modalities.

Textual data (lyrics and descriptions when available, as well as free-text tags) are lowercased, cleaned from boilerplate such as license notices, and tokenized with the transformer tokenizer used by the text encoder. Sequences longer than the maximum input length are truncated; shorter sequences are padded. Tracks without meaningful textual information are mapped to a special no-text token so that the fusion module can learn to down-weight the text modality in such cases.

Structured metadata, like artist IDs, genre labels, release year, and popularity measures (such as play counts), are normalized and encoded as explained in Section 3.3.3. Categorical data is turned into learned embeddings, while numbers are standardized and processed by a small neural network. If a dataset has a clear genre or mood system, like FMA and GTZAN, these labels are matched with the ontology classes and relationships from Section 3.5. This lets the same information be used both as input features and as rules for re-ranking.

For FMA, train, validation, and test sets are split by artist to make sure the same artist does not appear in more than one set. This prevents the task from being too easy. For GTZAN, standard balanced splits are used, and clips from the same original recording are kept together. In FSD50K, only music or music-related clips are chosen, and non-musical clips are used as hard negatives in some tests. MSD does not provide raw audio excerpts in the same form as FMA, GTZAN, and FSD50K. Therefore, it is not used for training raw-audio encoders from waveform segments. Instead, MSD contributes precomputed audio features and metadata representations and is included in the retrieval analysis in that form.

Positive and negative track pairs for contrastive training are chosen from these splits. Positive pairs are tracks with the same artist and genre, or different parts of the same track. Negative pairs come from different artists and genres, with more popular items chosen more often to match real catalogues. This approach helps the model group related items together and separate unrelated ones, following common practice in recommender systems.

4.3. Model Configurations and Training Setup

To answer the research questions from Section 3, the framework is set up in four different ways. Each setup uses the same preprocessing steps, encoder interfaces, fusion method, and ontology-based re-ranking function. The main differences are in which modalities are active and whether the ontology layer is used. This approach helps to clearly see how multimodal content and ontology guidance affect retrieval quality.

Configuration C1, which is the audio-only CNN baseline, uses a convolutional spectrogram encoder as explained in Section 3.3.1. The audio is first transformed into log-mel spectrograms, which are then processed by a stack of convolutional and pooling layers, followed by global pooling and a linear projection into the shared embedding space. This configuration represents a standard neural audio baseline that learns timbral and local temporal patterns directly from spectrogram representations. The resulting embedding for each track is based solely on audio content, and similarity is computed using cosine similarity as defined in Equation (1), without any ontology-based re-ranking.

Configuration C2 replaces the CNN backbone with a stronger pretrained audio encoder, while keeping the rest of the architecture unchanged. In the main reported setup, this configuration uses the published CLAP checkpoint laion/clap-htsat-unfused [27] as the audio backbone. The pretrained encoder output is followed by a shallow linear projection to the shared space of dimension d, and only this projection layer is trained on the similarity task. As in C1, ranking is based purely on cosine similarity between embeddings.

If comparative encoder analyses are performed, the same framework can also accommodate alternative pretrained music encoders, such as MERT [29] or MuLan [30], provided that their outputs are projected into the same shared representation space. However, the main reported experiments use CLAP as the default pretrained audio backbone.

In Configurations C3 and C4, the audio branch uses the same pretrained CLAP audio backbone as in C2. The configuration C3 uses multimodal fusion without ontology. It processes audio, text, and metadata with the encoders from Section 3.3 and the fusion method from Section 3.4. For each track, it creates embeddings for audio, text, and metadata, maps them to the same dimension, and combines them into one fused vector. Recommendation lists are made by finding the nearest neighbours using cosine similarity, as shown in Equation (1), without any changes based on ontology. This setup is used to measure the effect of multimodal fusion alone.

Configuration C4 corresponds to the full proposed framework: multimodal fusion followed by ontology-aware re-ranking. It uses the same multimodal encoders and fusion mechanism as C3, but additionally applies ontology-aware reranking via the score $S(q,i)$ defined in Equation (2), with the compatibility function ${\Phi }_{\mathrm{onto}}(q,i)$ given in Equation (3). The balance parameter λ and the ontology weights $w_{\mathrm{artist}},w_{\mathrm{genre}},w_{\mathrm{mood}}$ are tuned on a validation set. Comparing C4 to C3 isolates the contribution of the ontology layer for a fixed multimodal backbone.

In addition to Configurations C1–C4, this study includes an external published content-based retrieval baseline following classical MIR approaches based on handcrafted audio descriptors. In this instantiation, each track is represented using standard MIR features capturing complementary aspects of the signal, including timbre (MFCC), harmonic content (chroma), and spectral characteristics, aggregated into fixed-dimensional representations. The baseline is evaluated under the same retrieval protocol as the main framework and provides an external signal-level comparison outside the proposed C1–C4 design.

All configurations, including the external baseline, are evaluated under the same retrieval protocol to ensure comparability of results. Given a mini-batch of tracks, positive pairs are formed from items that share artist and genre labels or from different excerpts of the same track, while negatives are sampled from different artists and genres. A temperature-scaled NT-Xent loss, following standard practice in representation learning [39,40], encourages embeddings of positive pairs to have high cosine similarity and pushes negatives apart. Optimization is performed with the AdamW optimizer with decoupled weight decay [48], as implemented in PyTorch [42], using an initial learning rate of 1 × 10⁻⁴, weight decay of 1 × 10⁻⁴, and mini-batches of 64 pairs, unless otherwise stated. Early stopping on a held-out validation set is based on MAP@10 [41], ensuring that hyperparameters are aligned with the ranking objective used in the final evaluation.

The comparison among Configurations C1–C4 is intentionally controlled in order to isolate the contribution of stronger audio backbones, multimodal fusion, and ontology-guided reranking under a unified content-based retrieval protocol. In the revised evaluation, this controlled comparison is complemented by an external content-based reference baseline, allowing the proposed framework to be interpreted relative to both its internal ablations and to a conventional signal-level retrieval approach. The present results should still not be interpreted as a direct benchmark against interaction-driven recommender systems, graph-based recommenders, or sequence-aware playlist models, which typically rely on user-item logs and different evaluation settings.

4.4. Evaluation Protocol and Metrics

The primary evaluation task is content-based music retrieval. Given a query track q, the system ranks all candidate tracks i in the test catalogue according to the similarity score $S(q,i)$ defined in Section 3.6. For each configuration, embeddings are precomputed for all tracks in the test split and indexed using the same approximate nearest-neighbor backend as in Section 3.6, so that differences in performance reflect the learned representations and scoring functions rather than changes in the retrieval infrastructure.

Ground-truth relevance is determined by using the metadata described in Section 4.1. By default, a candidate track is considered relevant to a query if both the artist and genre label match. This criterion provides a practical and reproducible approximation of strong music-relatedness in public content-based datasets, especially in the absence of user-level similarity judgments. At the same time, this relevance definition partially overlaps with the ontology-aware similarity component used in Configuration C4, since the ontology term also incorporates artist- and genre-level agreement. Accordingly, the gains of C4 should be interpreted with appropriate caution: under the present protocol, they indicate stronger alignment with semantically defined retrieval targets, rather than constituting standalone evidence of deeper ontology reasoning. This issue is revisited in the limitations discussion.

For additional analysis, stricter and alternative relevance criteria are also considered, such as requiring the same artist together with a nearby release year, in order to examine how the configurations behave under more specific notions of similarity.

For a subset of experiments focused on binary genre-matching, I define a simpler label: a candidate is marked as relevant if its (possibly multi-label) genre annotation intersects with the query’s genre set, irrespective of the artist. These binary labels are used when analysing ROC and precision–recall curves.

At evaluation time, each track in the test split is used once as a query. For configurations, candidate tracks are ranked by cosine similarity in the embedding space. For Configuration C4, which includes ontology-guided reranking, candidates are ranked by the combined score from Equation (2), which blends embedding-based similarity with ontology-derived semantic compatibility. For each query, the system retrieves the top-k neighbours and compares the resulting ranked list against the relevance labels.

Retrieval performance is evaluated using standard ranking metrics from recommender systems and information retrieval [41]. The main focus is on the quality of the top part of the ranked list. Precision@10 measures the proportion of relevant tracks among the top 10 retrieved items for a query. Recall@10 measures the proportion of all relevant tracks for that query that appear within the top 10 retrieved items. For dataset-level evaluations and ablation analyses, F1@10 is computed at the query level as the harmonic mean of query-level Precision@10 and Recall@10, and is then averaged across queries. In the aggregated cross-dataset summary reported in

Table 3. Average retrieval performance of configurations C1–C4 across all datasets. For this aggregated cross-dataset summary, F1@10 is reported as the harmonic mean of the displayed average P@10 and R@10 values rather than as a separately bootstrapped query-level metric. For P@10, R@10, and MAP@10, 95% bootstrap confidence intervals based on 1000 resamples are shown in brackets.

Config	P@10	R@10	F1@10	MAP@10	ROC AUC	PR AUC
C1	0.708 [0.706, 0.710]	0.650 [0.647, 0.653]	0.678	0.730 [0.727, 0.733]	0.85	0.79
C2	0.738 [0.736, 0.740]	0.680 [0.677, 0.683]	0.708	0.750 [0.747, 0.753]	0.88	0.83
C3	0.770 [0.768, 0.772]	0.715 [0.712, 0.718]	0.741	0.780 [0.777, 0.783]	0.9	0.85
C4	0.780 [0.778, 0.782]	0.725 [0.722, 0.728]	0.751	0.790 [0.787, 0.793]	0.91	0.87

, however, F1@10 is shown as the harmonic mean of the displayed average P@10 and R@10 values for ease of comparison. In addition, Mean Average Precision at 10 (MAP@10) is reported: for each query, average precision is computed over the first ten ranked positions and then averaged across all queries, making MAP@10 sensitive to the exact ordering of relevant and non-relevant items near the top of the list.

Unless stated otherwise, Precision@10, Recall@10, F1@10, and MAP@10 are used as the main quantitative indicators, with scores averaged over all query tracks in the test split of each dataset. Additional experiments with other cutoffs, such as 5 and 20 retrieved items, showed similar relative ordering of the configurations, suggesting that the comparative trends are not strongly dependent on the exact cutoff.

To analyse threshold-dependent behavior in binary genre-matching scenarios, ROC and PR curves are also computed from the similarity scores between query and candidate tracks. For each configuration, relevance labels are treated as binary targets, and a decision threshold is swept over the similarity scores to obtain ROC and PR curves together with their corresponding areas under the curve (ROC AUC and PR AUC). These curves are used primarily for diagnostic analysis—for example, to compare configurations at low false-positive rates—and to support the aggregated AUC values reported in Section 4.5, rather than as the main optimization targets.

As an external reference point, the study additionally evaluates an external content-based MIR baseline based on standard audio descriptors widely used in the literature based on aggregated audio descriptors. The external baseline represents each track using a set of standard MIR audio descriptors capturing complementary aspects of the signal. In particular, timbral characteristics are modeled using Mel-frequency cepstral coefficients (MFCC), harmonic content is represented using chroma features, and additional spectral descriptors are included to capture properties such as brightness and spectral distribution. These frame-level features are aggregated over time to obtain a fixed-dimensional representation for each track. Similarity between tracks is then computed using cosine similarity in this feature space, and retrieval is performed by ranking candidates according to this similarity score. Candidate tracks are ranked using cosine similarity under the same retrieval protocol as Configurations C1–C4. This baseline serves as a reproducible signal-level comparator outside the proposed multimodal and ontology-guided framework.

4.5. Aggregated Retrieval Results Across Datasets

This subsection summarizes the retrieval performance of Configurations C1–C4 across all datasets described in Section 4.1. The goal is to compare audio-only, multimodal and ontology-aware variants under a unified evaluation protocol and to assess how robust these patterns are across different catalogues and annotation regimes.

The dataset-level retrieval results show a consistent pattern across configurations. On FMA, where rich textual metadata and a hierarchical genre taxonomy are available, the gains from C1 and C2 to C3 and C4 are the most pronounced: the multimodal model already brings a substantial boost over the audio-only baselines, and the ontology-aware variant further improves both MAP@10 and F1@10. On MSD, which is larger and annotated with noisier tags, C2 consistently outperforms C1, while C3 and C4 provide additional but more moderate gains. The relatively small gap between C3 and C4 suggests only a modest additional benefit of ontology-guided reranking under the present protocol, rather than a large shift in retrieval quality. On GTZAN, a small genre-only dataset with limited annotations, all configurations reach relatively high scores and the differences between C2, C3, and C4 remain modest, indicating that audio-only representations already capture most of the discriminative signal. For FSD50K, which focuses on general sound events rather than music, the music-centric ontology used in C4 does not align with the underlying label space, so C3 and C4 behave very similarly in terms of MAP@10 and F1@10, and ontology-aware reranking brings little additional benefit beyond the multimodal baseline.

Table 3 reports the corresponding average retrieval scores of C1–C4 over MSD, FMA, FSD50K and GTZAN. Across all metrics, the same trend appears consistently: the simple audio-only baseline (C1) achieves the lowest scores, the stronger pretrained audio model (C2) provides a noticeable improvement, the multimodal configuration (C3) further improves average P@10, F1@10, and MAP@10, and the ontology-aware hybrid configuration (C4) attains the highest average scores, although the margin over C3 is modest in some cases and should be interpreted cautiously. In other words, as more information is made available to the system—first audio, then audio plus text, and finally ontology-guided structure—the average top-ranked results tend to improve as additional information sources are introduced, although the incremental gain from C3 to C4 remains modest in some settings. The following table presents an aggregated cross-dataset summary intended to illustrate the overall pattern across configurations; the individual dataset-level comparisons remain the primary basis for interpretation.

In the aggregated summary, F1@10 is included as a descriptive harmonic-mean summary of the displayed average precision and recall values, whereas bootstrap confidence intervals are reported for P@10, R@10, and MAP@10.

Bootstrap confidence intervals (95%, 1000 resamples) are reported in Table 3 for the main retrieval metrics. The improvement from C3 to C4 is small in absolute magnitude. For MAP@10, however, the corresponding bootstrap intervals do not overlap, suggesting that the gain is modest yet consistent under the present evaluation protocol. ROC AUC and PR AUC are included as supplementary diagnostic measures and are shown without bootstrap intervals.

To complement the aggregated internal comparison in Table 3, an external published content-based MIR baseline was additionally evaluated under the same retrieval protocol. This conventional MIR baseline, based on signal-level audio descriptors, achieved P@10 = 0.222, R@10 = 0.148, F1@10 = 0.120, and MAP@10 = 0.389 on FMA, which serves as the primary benchmark with full access to raw audio features. As expected, this external signal-level baseline performs below the stronger learned and multimodal configurations, but it provides a meaningful reference point beyond internal ablations alone.

Systems that use multiple information sources consistently perform better than those using only audio. Adding ontology-aware re-ranking gives further improvement when the dataset has detailed and compatible metadata, as seen in FMA and, to a lesser degree, MSD. However, on GTZAN, where metadata are limited, and on FSD50K, where labels refer to general sound events instead of musical concepts, the hybrid model performs similarly to the multimodal baseline. Taken together with the externally published MIR baseline, these findings suggest that the framework provides gains beyond a conventional signal-level retrieval reference, while the advantages of the ontology appear mainly when the catalogue and its annotations fit the structure in the knowledge graph.

To better understand which design choices underlie these cross-dataset differences, Section 4.6 examines the contribution of modality fusion, ontology weighting, and encoder choice in more detail.

4.6. Comparative Observations on Modality, Ontology Weighting, and Encoder Choice

This subsection examines how individual components of the framework contribute to retrieval performance under the shared evaluation protocol from Section 4.4. Three complementary analyses are reported. First, a modality ablation evaluates the incremental contribution of text and metadata beyond the audio branch. Second, a sensitivity analysis over the ontology balance parameter λ examines the trade-off between retrieval quality and diversity-related indicators. Third, a focused backbone comparison assesses the effect of replacing the lightweight CNN baseline with the default pretrained CLAP-based audio backbone. Together, these analyses clarify which parts of the framework are responsible for the observed gains and under what conditions they are most effective.

4.6.1. Modality Contribution

To assess the contribution of each modality, the framework is evaluated under four settings: audio only, audio plus text, audio plus metadata, and full multimodal fusion without ontology-guided reranking.

Table 4. Modality ablation results on the main benchmark setting (e.g., FMA). F1@10 is averaged over query-level F1 values.

Audio	Text	Metadata	P@10	R@10	F1@10	MAP@10
+	−	−	0.72	0.66	0.69	0.73
+	+	−	0.74	0.67	0.7	0.75
+	−	+	0.75	0.7	0.72	0.76
+	+	+	0.77	0.71	0.74	0.78

reports the results on the main benchmark setting.

Table 4 indicates that the audio-only setting yields the weakest retrieval quality, while the addition of text and metadata improves performance incrementally. The gains are moderate rather than dramatic, suggesting that multimodal fusion refines the core audio similarity signal rather than replacing it. In semantically rich datasets such as FMA, the effect is clearest because textual descriptions and structured metadata provide complementary cues that cannot be recovered from audio alone. By contrast, in weaker or noisier metadata regimes, the same progression is expected to remain more limited.

4.6.2. Sensitivity to the Ontology Balance Parameter

The effect of ontology-guided reranking is analysed by varying the balance parameter λ in Equation (2) from 0 to 1. Here, λ = 0 corresponds to embedding-only ranking, whereas λ = 1 corresponds to ontology-dominated ranking. Intermediate values define hybrid scoring regimes that combine both sources of information.

Table 5. Sensitivity of retrieval quality and diversity to the ontology balance parameter λ on the main benchmark setting. F1@10 is averaged over query-level F1.

λ	P@10	R@10	F1@10	MAP@10	Long-Tail Share@10
0	0.77	0.72	0.74	0.78	0.21
0.25	0.78	0.73	0.75	0.79	0.23
0.5	0.78	0.74	0.76	0.8	0.26
0.75	0.77	0.72	0.74	0.78	0.28
1	0.74	0.7	0.72	0.76	0.31

summarizes the resulting retrieval and diversity-related trends.

Table 5 suggests that moderate values of λ provide the most favorable balance between retrieval quality and long-tail exposure. As ontology weighting increases, long-tail share rises steadily, whereas MAP@10 peaks at intermediate values and declines under ontology-dominated settings. This pattern is consistent with the interpretation that ontology-guided reranking is most useful when it complements the embedding-based similarity signal rather than replacing it entirely. In semantically aligned settings such as FMA, moderate ontology weighting can refine the neighborhood structure, whereas extreme weighting shifts the ranking too strongly toward symbolic agreement.

4.6.3. Effect of Replacing the CNN Baseline with the CLAP-Based Audio Backbone

To assess the contribution of the audio representation itself, the audio backbone is changed from the lightweight convolutional baseline to the default pretrained CLAP-based audio backbone used in the main experiments. This comparison isolates the effect of stronger audio representation learning relative to the simpler baseline.

Table 6. Effect of replacing the CNN baseline with the CLAP-based audio backbone. F1@10 is averaged over query-level F1 values.

Audio Backbone	Pretrained	P@10	R@10	F1@10	MAP@10
CNN	No	0.74	0.69	0.71	0.75
CLAP	Yes	0.79	0.74	0.76	0.8

indicates that replacing the lightweight convolutional baseline with the pretrained CLAP-based audio backbone improves the absolute level of retrieval performance. This suggests that encoder strength contributes materially to the quality of the base audio representation. In turn, it supports the broader interpretation that part of the improvement from C1 to C2 in Section 4.5 is attributable to stronger pretrained audio features rather than to changes in the downstream ranking framework alone.

Taken together, these analyses are consistent with the main interpretation of the framework: audio representations provide the core similarity signal, multimodal fusion improves retrieval when semantic side information is available, and ontology-guided reranking contributes additional gains when the semantic layer is well aligned with the catalogue structure.

4.7. Embedding Space and Diversity Visualisation

The quantitative results in Section 4.4, Section 4.5 and Section 4.6 characterize how the configurations differ in ranking metrics but do not show how these differences manifest in the internal structure of the learned representations and in the composition of the recommended lists. This subsection, therefore, complements the numerical evaluation with visual analyses of the embedding space and of diversity-related properties of the top-ranked results.

Figure 2 illustrates the geometry of the embedding space on a genre-labelled subset of the catalogue. For visualisation, the track embeddings were projected to two dimensions using UMAP [49] with n_neighbors = 15, min_dist = 0.1, metric = cosine, and random_state = 42. The same projection settings were used across configurations to make the qualitative comparison more consistent. Two-dimensional projections are shown for three configurations: the audio-only baseline (C1), the multimodal configuration (C3), and the ontology-aware hybrid variant (C4). In the audio-only case, genres that are acoustically similar tend to form overlapping or elongated clusters, and neighbourhoods often mix tracks from different artists and loosely related genres. In the multimodal configuration, the clusters become more compact and genre boundaries are easier to distinguish, reflecting the influence of textual descriptors and metadata. The ontology-aware variant further sharpens these patterns: clusters corresponding to closely related genres are better separated, while cross-genre regions appear in a more interpretable way, typically grouping tracks that share mood, instrumentation, or other high-level properties rather than only surface-level acoustic similarity.

Figure 3 looks at two basic diversity measures: the average number of different artists in the top ten recommendations (artist coverage@10) and the share of less popular tracks among these recommendations (long-tail share@10). The plots display these values for configurations C1 to C4 on FMA and MSD. In both datasets, artist coverage and long-tail share rise from C1 to C4, with the biggest increases in the ontology-aware setup. On FMA, this greater diversity comes with clear improvements in MAP@10 and F1@10, showing that more long-tail content does not reduce relevance. On MSD, the same pattern appears but is less strong, likely due to messier and less organized metadata. On GTZAN and FSD50K, which have limited metadata or use labels that do not match the music-focused ontology, changes in these measures are small and match the smaller differences in overall retrieval performance mentioned earlier.

Taken together, these visualisations suggest that ontology-aware re-ranking does not merely adjust scores in an otherwise fixed space, but reshapes local neighbourhoods to better reflect high-level musical structure and increase the visibility of less prominent artists and tracks. This behaviour is most evident in datasets with rich, compatible metadata and provides additional support for the strengths and limitations of the proposed framework beyond what can be seen from ranking metrics alone.

5. Discussion

This study started from a simple assumption: assessing music similarity should not rely on audio alone, but should also make use of textual descriptions and structured musical knowledge [3,5]. The experiments on MSD, GTZAN, FSD50K and FMA [44,45,46,47] are broadly consistent with this assumption. Configurations that use only audio embeddings systematically achieve the lowest MAP@10 and F1@10 scores, and adding more information gradually improves the results. Moving from a basic audio baseline to a stronger pretrained encoder already gives a noticeable gain, and adding text and metadata on top of that provides a further improvement on all datasets. The effect is clearest on FMA, which offers detailed descriptions and fine-grained genre labels, and more moderate on MSD with its noisy tags. This behavior is in line with earlier work on content-driven and multimodal recommendation, where semantic context is shown to complement purely acoustic features [5,13,27].

The ontology adds another layer to this picture. When the structure of the catalogue and the ontology are reasonably aligned—as on FMA and, to a lesser extent, MSD—the hybrid configuration performs better than both the audio-only and purely multimodal variants. The benefit appears not only in top-10 ranking metrics, but also in how the embedding space is organised and which tracks are returned as neighbours. Using a graph of musical concepts follows prior work on music and sound ontologies [35,38] and hybrid graph-based recommenders [18,19], but here the graph is used purely as a re-ranking signal on top of multimodal embeddings. In this setting, ontology-guided similarity tends to pull together tracks that are musically related but acoustically borderline, and to separate clusters that would otherwise be mixed by surface-level features. At the recommendation level, this translates into higher artist coverage and a larger share of long-tail tracks in the top results, especially on FMA, while precision and MAP@10 remain at a competitive level. Quantitative evidence for these diversity-related effects is reported in Figure 3 through artist coverage@10 and long-tail share@10, and in Table 5 through the sensitivity analysis of ontology weighting and long-tail exposure.

At the same time, this result should be interpreted with appropriate caution. In the present evaluation protocol, relevance is defined primarily through artist and genre agreement, and these attributes are also available to the ontology-aware component used in Configuration C4. As a result, part of the observed improvement may reflect stronger alignment with semantically defined relevance labels rather than independent evidence of deeper ontology reasoning alone. This does not invalidate the comparison, since all configurations are evaluated under the same protocol, but it does limit how strongly the C4 gains should be interpreted. In this sense, the hybrid model is best understood as improving alignment with structured semantic retrieval targets under the current benchmark.

The ontology-guided layer also makes the similarity score easier to interpret and inspect. The final ranking is based on two parts: similarity in the learned embedding space and semantic compatibility from the ontology. This makes it possible to generate simple concept-level explanations for each recommendation by showing what comes from the audio/text match and what comes from shared attributes such as artist, genre, or mood. In practice, this supports simple concept-level explanations such as: it is recommended because it sounds similar and shares genre X or mood Y. It is also useful for debugging, since we can see when a strong embedding match is reduced by a semantic mismatch, or when semantic agreement improves the ranking.

Equally important are the cases where the ontology has little effect. GTZAN is small and offers only coarse genre labels, so the stronger audio baselines, the multimodal model and the hybrid configuration end up with very similar scores [45]. In this regime, most of the signal is already captured by audio, and there is little additional structure for text or the ontology to exploit. FSD50K shows a different mismatch: its labels describe general sound events, while the ontology is centred on musical concepts [38,46]. In this case, the hybrid configuration closely tracks the multimodal baseline in MAP@10, F1@10 and AUC-based metrics. The ontology neither helps nor harms. Together, these examples point to a simple condition: ontology-aware re-ranking is useful only when there is a dense and reliable mapping between catalogue items and the concepts in the graph; when this mapping is weak, the extra layer is largely neutral [18,35,38].

Additional comparative observations support this interpretation from another angle. When the weight of the ontology-based component is varied, a familiar trade-off appears: increasing the contribution of the ontology tends to improve ranking quality and diversity up to a point, after which the gains saturate and precision starts to drop [41]. A similar pattern holds for the audio backbone. Replacing a lightweight convolutional encoder with more powerful pretrained audio and audio–text models consistently raises the absolute scores, but does not change the relative ordering of configurations [17,27,29]. Multimodal variants remain stronger than audio-only baselines, and the ontology-aware model remains the strongest observed option where the metadata and ontology are compatible. This suggests that future advances in representation learning may improve the overall level of performance further, while the role of ontology alignment and data structure is likely to remain important [5,18,19].

The choice of evaluation datasets is intentionally narrow. MSD, GTZAN, FSD50K and FMA [44,45,46,47] cover different combinations of catalogue size, annotation richness and domain, and are sufficient to expose the main strengths and limitations of the proposed framework without diluting the analysis over many corpora. This also means that the present study should be read primarily as a controlled content-based evaluation rather than as a direct comparison with interaction-driven recommender systems. At the same time, several public collections have become standard in music recommendation and are natural candidates for extending the evaluation. MagnaTagATune is widely used for automatic tagging and multi-label classification and provides dense clip-level annotations [50]. Last.fm-based datasets such as Last.fm −1K and Last.fm-360K offer large-scale listening histories and user profiles [6,51]. The Spotify Million Playlist Dataset is a reference benchmark for playlist continuation and sequence-aware recommendation [20,52]. More recently, the #nowplaying-RS dataset has been proposed as a context-aware benchmark that combines listening events with item content and user context features [25,53]. Applying the hybrid multimodal–ontology framework to these user- and playlist-centric collections would make it possible to test its behaviour under different interaction patterns and to compare it more directly with collaborative and sequence-based approaches [18,21].

One of the limitations of the current revision is that it does not include formal confidence intervals or significance tests for all configuration differences in the main results tables. Accordingly, small gaps such as those between C3 and C4 should be interpreted cautiously, especially when the absolute differences in MAP@10 or related ranking metrics are modest.

Another limitation of the present study is the absence of broader external baseline comparisons. The empirical analysis is intentionally centered on controlled comparisons among internal configurations of the proposed framework, so that the effects of encoder strength, multimodal fusion, and ontology-guided reranking can be isolated under a shared retrieval protocol. The goal of the present study is therefore not to establish a new state of the art against all recommender families, but to analyze in a controlled way how the proposed components affect content-based retrieval behavior. As a result, the study does not yet provide a direct empirical comparison against collaborative filtering, graph-based recommendation, or sequence-aware playlist models, which typically require different datasets, user-interaction logs, and evaluation settings. The ontology component in the present study should therefore be interpreted as a lightweight ontology-guided semantic layer for interpretable reranking rather than as a full formal ontology with hierarchical reasoning. This limitation is partially addressed in the revised manuscript by adding an external published content-based MIR baseline based on conventional MIR audio descriptors widely used in prior work [54,55]. Although this baseline is simpler than the learned multimodal configurations, it provides a reproducible external signal-level comparator and helps position the proposed framework beyond internal ablations alone.

Several directions for further work follow naturally. The current ontology mostly covers genres, broad mood categories and a limited set of musical attributes [35,38]. This also reflects a current limitation of the study: the semantic layer remains deliberately lightweight and does not yet model richer hierarchies, subgenre relations, or graded concept similarity. Extending it towards instrumentation, production characteristics and cultural context, and improving automatic alignment between graph nodes and catalogue items, could amplify the gains seen on FMA and MSD and potentially help on datasets where the benefit is currently small. The evaluation is also restricted to item–item similarity; user histories, temporal dynamics and business constraints are not modelled explicitly. Likewise, the present experiments do not include user-centered evaluation of explanation quality, faithfulness checks, or broader behavioral outcomes. Integrating the similarity layer into a full recommendation pipeline would make it possible to study how ontology-aware re-ranking interacts with personalisation, exposure control and novelty objectives [6,9,41]. Another promising direction is to replace the fixed balance between embedding-based and ontology-based similarity with an adaptive scheme that depends on the query, on local data density, or on the reliability of metadata [18,38]. Finally, the diversity analysis in this study relies on simple indicators such as artist coverage and long-tail share [6,7,11]. More refined measures of novelty, serendipity and fairness could give a richer picture of how graph-guided similarity affects listener experience and catalogue exposure, particularly on user-centric datasets [9,11].

The clearest improvements are observed when the data’s semantic structure matches the concepts in the ontology. When this alignment is weak, the framework behaves much more like a strong multimodal baseline. This reliance on metadata quality and ontology alignment is both a limitation and a guide for design. In real-world use, the value of these models depends as much on the catalogue and its annotations as on the neural architecture itself [5,6,18]. Future research should focus on designing model architectures, ontologies, and catalog curation practices together, instead of treating them as separate parts.