Novelty metrics

The novelty of a recommendation measures to which extent the user was unaware of the recommended people in the network. RELISON includes the following novelty measures:

Long tail novelty
Mean prediction distance
Unexpectedness

Long tail novelty

This metric (also known as expected popularity complement) represents the prior probability that a random person in the network did not know about the recommended people:

\[\mbox{LTN} = \frac{1}{|\hat{E}|} \sum_{(u,v)\in\hat{E}}\left(1 - \frac{|\Gamma_{in}(v)|}{|\mathcal{U}|}\right)\]

References:

1. Castells, N. J. Hurley, and Saúl Vargas. Novelty and Diversity in Recommender Systems. Recommender Systems Handbook, 2nd ed., 881–918 (2015).
1. Vargas and P. Castells. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. 5th ACM Conference on Recommender Systems (RecSys 2011), 109-116 (2011)

Input

In order for this metric to work, the recommendation program must receive a community partition.

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

LTN:
  cutoff:
    type: int
    values: [1,5,10]

Mean prediction distance

This metric measures how far from the target user the recommended people are. It is computed as the harmonic mean of the reciprocal distances between the target and recommended users:

\[\mbox{MPD} = \frac{|\hat{E}|}{\sum_{(u,v)\in\hat{E}} \frac{1}{\delta(u,v)}} - 2\]

References:

1. Sanz-Cruzado, S.M. Pepa, P. Castells. Structural novelty and diversity in link prediction. 9th International Workshop on Modeling Social Media (MSM 2018) at The Web Conference (WWW 2018). The Web Conference Companion, pp. 1347–1351.
1. Sanz-Cruzado, P. Castells. Beyond Accuracy in Link Prediction. BIAS 2020: Bias and Social Aspects in Search and Recommendation, pp 79-94.

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

Mean prediction distance:
  cutoff:
    type: int
    values: [1,5,10]

Unexpectedness

This metric measures how different the recommended users are to the contacts the target user already has:

\[\mbox{Unexp} = \frac{1}{\hat{E}} \sum_{(u,v)\in\hat{E}} \frac{1}{|\Gamma_{out}(u)|} \sum_{w \in \Gamma_{out}(u)} d(v,w)\]

where \(\hat{E}\) represents the whole set of recommended links, and \(d(u,v)\) is the distance between the users (in this case, represented by one minus the cosine of the feature vector for each user).

\[p(c|u) = \frac{|\{v \in c | (u,v) \in E \cup E_{test}\}|}{\sum_{c'\in \mathcal{C}} |\{v \in c' | (u,v) \in E \cup E_{test}\}|}\]

References:

1. Castells, N. J. Hurley, and Saúl Vargas. Novelty and Diversity in Recommender Systems. Recommender Systems Handbook, 2nd ed., 881–918 (2015).
1. Vargas and P. Castells. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. 5th ACM Conference on Recommender Systems (RecSys 2011), 109-116 (2011)

Input

In order for this metric to work, the recommendation program must receive a set of features representing the users (or an index, generated by the TwittomenderIndexGenerator program).

Parameters

cutoff: the (maximum) number of recommended users to consider in the computation of the metric.

Configuration file

Unexpectedness:
  cutoff:
    type: int
    values: [1,5,10]