Diversity metrics

Beyond the accuracy of the recommendations, one of the main dimensions which is commonly studied is their diversity (i.e. how different the recommended users are among them). RELISON includes the following diversity measures:

Community recall
ERR-IA
Intra-list diversity
Predicted Gini complement

Community recall

The community recall is a metric based on the subtopic recall diversity metric in information retrieval. It is defined as the proportion of different communities the target user could reach through a direct link if he followed all the recommended users:

\[\mbox{CommRecall}(u) = \frac{1}{|\mathcal{C}}\left|\bigcup_{v \in R_u} c(v)\right|\]

References:

1. Zhai, W. W. Cohen, and J. Lafferty. Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval. 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2003), 10–17 (2003).
1. Sanz-Cruzado, S.M. Pepa, P. Castells. Structural novelty and diversity in link prediction. 9th International Workshop on Modeling Social Media (MSM 2018) at The Web Conference (WWW 2018). The Web Conference Companion, pp. 1347–1351.
1. Sanz-Cruzado, P. Castells. Beyond Accuracy in Link Prediction. BIAS 2020: Bias and Social Aspects in Search and Recommendation, pp 79-94.

Input

In order for this metric to work, the recommendation program must receive a community partition.

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

Community recall:
  cutoff:
    type: int
    values: [1,5,10]

ERR-IA

This metric combines the accuracy of the recommendation with measuring the different aspects retrieved by the recommendation (here, we take communities as aspects). This particular metric, ERR-IA, given a correctly predicted link, it weights down its importance when the community of the recommended candidate has already appeared in previous positions of the ranking. It is defined as:

\[\mbox{ERR-IA}(u) = \sum_{c\in\mathcal{C}} p(c|u) \mbox{ERR-IA}(u,c)\]

and

\[\mbox{ERR-IA}(u,c) = \sum_{k=1}^{|R_u|} \left[ \frac{1}{k} p(\mbox{rel}|v_k,c) \prod_{j=1}^{k-1} \left(1 - p(\mbox{rel}|v_j,c)\right)\right]\]

where \(v_k\) is the recommended user in the \(k\)-th position in the ranking, and:

\[p(\mbox{rel}|v,c) = \frac{1}{2} 1_{\{(u,v) \in E_{test} \wedge c(v) = c\}} (u,v)\]

where \(E_{test}\) represents the set of test links. The probability \(p(c|u)\) represents the probability that the user \(u\) is interested on community \(c\), and it is estimated by:

\[p(c|u) = \frac{|\{v \in c | (u,v) \in E \cup E_{test}\}|}{\sum_{c'\in \mathcal{C}} |\{v \in c' | (u,v) \in E \cup E_{test}\}|}\]

Reference: O. Chapelle, S. Ji, C. Liao, E. Velipasaoglu, L. Lai, and S. Wu. Intent-based diversification of web search results: metrics and algorithms. Information Retrieval, 14(6), 572–592 (2011)

Input

In order for this metric to work, the recommendation program must receive a community partition.

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

nDCG:
  cutoff:
    type: int
    values: [1,5,10]

Intra-list diversity

The intra-list diversity of a recommendation, measures the average distances between the recommended users. For this, it uses some feature data (for instance, the contents of the tweets published by a user).

\[\mbox{ILD} = \frac{1}{|\hat{E}|} \sum_{(u,v)\in\hat{E}} \sum_{w \in R_u} \frac{d(v,w)}{|R_u|}\]

where \(\hat{E}\) represents the whole set of recommended links, \(R_u\) the set of users recommended to user \(u\), and \(d(u,v)\) is the distance between the users (in this case, represented by one minus the cosine of the feature vector for each user).

References:

1. Castells, N. J. Hurley, and Saúl Vargas. Novelty and Diversity in Recommender Systems. Recommender Systems Handbook, 2nd ed., 881–918 (2015).
1. Vargas and P. Castells. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. 5th ACM Conference on Recommender Systems (RecSys 2011), 109-116 (2011)

Input

In order for this metric to work, the recommendation program must receive a set of features representing the users (or an index, generated by the TwittomenderIndexGenerator program).

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

ILD:
  cutoff:
    type: int
    values: [1,5,10]

Predicted Gini complement

The predicted Gini complement how balanced the distribution of recommended users is among all the produced recommendations. It uses for this the Gini complement:

\[\mbox{PGC} = 1 - \frac{1}{|\mathcal{U}|-1} \sum_{i=1}^|\mathcal{U}| (2i - |\mathcal{U}| - 1)\frac{|\{u | v_i \in R_u\}|}{\sum_{u} |R_u|}\]

where \(R_u\) represents the recommendation ranking for user \(u\), and \(v_i\) is the \(i\)-th less recommended user.

References:

1. Vargas and P. Castells. Improving Sales Diversity by Recommending Users to Items. 8th ACM Conference on Recommender Systems (RecSys 2014), 145–152 (2014).

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

Predicted Gini complement:
  cutoff:
    type: int
    values: [1,5,10]