{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "LightFM is a very popular tool, which finds its applications both in small projects and in production systems. On a high level, the general idea of representing users and items in terms of combinations of their features, used in the LightFM model, is not new. It was implemented in a number of preceeding works, and I'm not even saying about the obvious link to [Factorization Machines](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf). Another notable member of this family is [SVDFeature](https://arxiv.org/pdf/1109.2271.pdf) , which came around in 2011 and in some sense was even closer to LightFM than FM (the naming, though, is misleading, as it is not an SVD-based model). In fact, the feature combination approach can be traced back to the Microsoft's [MatchBox system](https://www.microsoft.com/en-us/research/wp-content/uploads/2009/01/www09.pdf) from 2009. The authors used the term *trait vector* for a combined representation and employed elaborate probabilistic framework on top of it. Still, even though the idea itself was not new, the LightFM model was made convenient and extremely easy to use, which, probably, has made this framework popular among practitioners. Moreover, it provided several ranking optimization schemes and had a fairly good computational performance.\n", "\n", "On the other hand, I have also seen some complains from data scientists about actual prediction quality of LightFM. I could not find a more or less rigorous evaluation of LightFM, where it would be compared against strong baselines and, more importantly, all models would undergo appropriate tuning. The [examples section](https://lyst.github.io/lightfm/docs/examples.html) of the LightFM's documentation merely provides quick-start demos, not real performance tests. The [paper from the RecSys workshop](http://ceur-ws.org/Vol-1448/paper4.pdf) also does not provide too much evidence on the model's performance. Numerous online tutorials only repeat basic configuration steps, leaving the whole tuning aspect aside. Considering how often practitioners are advised to start with LightFM (in my experience), I have decided **to put the LightFM capabilities to the real test**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n",
"\n",
"
| \n",
" \n",
"\n",
"
| \n",
" \n",
"\n",
"
| \n",
"
\n",
"\n",
"
| \n",
" \n",
"\n",
"
| \n",
"
\n", " | precision | \n", "item_alpha | \n", "rank | \n", "epochs | \n", "metrics | \n", "
---|---|---|---|---|---|
92 | \n", "0.033974 | \n", "2.986615e-07 | \n", "6 | \n", "100 | \n", "[(0.03397402597402597, 0.25204923647780786, No... | \n", "
99 | \n", "0.034078 | \n", "1.901221e-06 | \n", "12 | \n", "100 | \n", "[(0.034077922077922075, 0.25326630512344794, N... | \n", "
53 | \n", "0.034078 | \n", "5.043306e-06 | \n", "25 | \n", "25 | \n", "[(0.034077922077922075, 0.2526167166167166, No... | \n", "
60 | \n", "0.034234 | \n", "3.091518e-09 | \n", "9 | \n", "50 | \n", "[(0.034233766233766234, 0.2541265401265401, No... | \n", "
91 | \n", "0.034390 | \n", "7.262623e-07 | \n", "9 | \n", "100 | \n", "[(0.034389610389610394, 0.2558374958374958, No... | \n", "
col_scaling | -0.8 | -0.6 | -0.4 | -0.2 | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 |
---|---|---|---|---|---|---|---|---|---|
rank | |||||||||
1 | \n", "0.0224 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "0.0231 | \n", "
15 | \n", "0.0255 | \n", "0.0253 | \n", "0.0257 | \n", "0.0261 | \n", "0.0259 | \n", "0.0259 | \n", "0.0260 | \n", "0.0263 | \n", "0.0269 | \n", "
30 | \n", "0.0266 | \n", "0.0267 | \n", "0.0269 | \n", "0.0268 | \n", "0.0270 | \n", "0.0268 | \n", "0.0282 | \n", "0.0282 | \n", "0.0275 | \n", "
45 | \n", "0.0294 | \n", "0.0289 | \n", "0.0290 | \n", "0.0290 | \n", "0.0291 | \n", "0.0290 | \n", "0.0290 | \n", "0.0298 | \n", "0.0288 | \n", "
60 | \n", "0.0286 | \n", "0.0286 | \n", "0.0287 | \n", "0.0289 | \n", "0.0289 | \n", "0.0292 | \n", "0.0286 | \n", "0.0285 | \n", "0.0287 | \n", "
75 | \n", "0.0279 | \n", "0.0279 | \n", "0.0281 | \n", "0.0283 | \n", "0.0285 | \n", "0.0287 | \n", "0.0286 | \n", "0.0286 | \n", "0.0291 | \n", "
90 | \n", "0.0279 | \n", "0.0276 | \n", "0.0276 | \n", "0.0277 | \n", "0.0278 | \n", "0.0282 | \n", "0.0279 | \n", "0.0283 | \n", "0.0285 | \n", "
105 | \n", "0.0279 | \n", "0.0279 | \n", "0.0284 | \n", "0.0282 | \n", "0.0284 | \n", "0.0283 | \n", "0.0285 | \n", "0.0287 | \n", "0.0286 | \n", "
120 | \n", "0.0279 | \n", "0.0279 | \n", "0.0276 | \n", "0.0276 | \n", "0.0281 | \n", "0.0282 | \n", "0.0282 | \n", "0.0281 | \n", "0.0282 | \n", "
135 | \n", "0.0270 | \n", "0.0271 | \n", "0.0272 | \n", "0.0274 | \n", "0.0273 | \n", "0.0273 | \n", "0.0276 | \n", "0.0276 | \n", "0.0281 | \n", "
150 | \n", "0.0266 | \n", "0.0268 | \n", "0.0272 | \n", "0.0273 | \n", "0.0273 | \n", "0.0276 | \n", "0.0276 | \n", "0.0276 | \n", "0.0278 | \n", "
165 | \n", "0.0264 | \n", "0.0269 | \n", "0.0266 | \n", "0.0267 | \n", "0.0268 | \n", "0.0275 | \n", "0.0273 | \n", "0.0274 | \n", "0.0270 | \n", "
180 | \n", "0.0256 | \n", "0.0257 | \n", "0.0261 | \n", "0.0263 | \n", "0.0261 | \n", "0.0262 | \n", "0.0266 | \n", "0.0268 | \n", "0.0274 | \n", "
195 | \n", "0.0252 | \n", "0.0251 | \n", "0.0253 | \n", "0.0256 | \n", "0.0254 | \n", "0.0258 | \n", "0.0263 | \n", "0.0260 | \n", "0.0261 | \n", "
precision | recall | miss_rate | nDCG | coverage | |
---|---|---|---|---|---|
SVD (best) | \n", "0.0228118 | \n", "0.19749 | \n", "0.80251 | \n", "0.130738 | \n", "0.016765 | \n", "
LightFM (best) | \n", "0.0322757 | \n", "0.285704 | \n", "0.714296 | \n", "0.20512 | \n", "0.190934 | \n", "
HybridSVD (rank 135) | \n", "0.0374726 | \n", "0.328895 | \n", "0.671105 | \n", "0.237991 | \n", "0.104315 | \n", "