Why good models sometimes make bad recommendations (a video!)

I’m an organizer of A.I. Sorcatic Circles, a machine learning discussion group based in Toronto, focused on reviewing highly technical advances in machine learning literature.

We hold events one to two times a week, and stream our sessions online on our YouTube channel. Our sessions run about an hour and a half long, so this makes for content that’s very long. As part of an effort to produce shorter, more digestible content for YouTube, we came up with The 5 Minute Paper Challenge.

Participating in The 5 Minute Paper Challenge is simple in its description, but complicated in its execution. Simply put, participants were asked to create videos of 5 minutes in length, describing their favourite machine learning paper. It may sound easy, but it really isn’t. I learned the hard way, through spending a few days putting together a “sample” video for our participants to refer to. I thought it’d be a matter of writing a script, pointing a camera at my face, and quick editing. I was wrong.

Take a look at my video below, if you’d like. The paper I chose to explain is called Folding: Why Good Models Sometimes Make Spurious Recommendations. It discusses how matrix-based collaborative filtering systems often embed unrelated groups of users close to each other in the embedding space. The way to rectify it, they say, is to use some sort of goodness metric (like RMSE) along with some badness metric (they propose a “folding” metric), when training your model and evaluating its performance.

I have to be honest, I’m pretty proud of what I came up with! And it was a lot of fun.

A few problems

I mentioned that I learned “the hard way” how difficult it is to creating a 5 minute-long video explaining a machine learning paper. I’ll list a few of the difficulties below:

  • Explaining something you understand well to an audience that’s learning something for the first time is difficult to do with any amount of time, let alone with a 5 minute time constraint. Added on top of that constraint is the requirement to provide a sufficient introduction, algorithm description, and discussion of results, all in 5 minutes. So everything you say had better be crystal clear. Fortunately, this is something that can be improved with practice, and by sharing your explanation with a friend and getting and incorporating their feedback.
  • Editing is not as simple as it looks. Especially when you’ve never edited before! For me, this was the bottle neck step in my content creation process. Next time I know I’ll be a lot faster and more comfortable.
  • Filming yourself is also not as simple as it looks. It’s difficult to get the audio and video quality (the focus) right when it’s just you. To improve this video I would use a different mic (maybe I could record myself on my phone while I film, and use that audio for the final video), and I’d possibly ask someone to help film, so they can make sure that my face stays in focus.
  • Also about filming, it’s important to make sure the lighting in the room is reproducible, in case you need to re-film. I didn’t realize this until it was too late, but it’s easy to fix for next time.

The Challenge, round 2? And, the community’s content

Now although it was a bit tough, it was an extremely enjoyable and rewarding process, and I’m excited to do it again when we host the next 5 Minute Challenge. I’m hoping I’ll be able to take the lessons I learned in this first iteration, and apply those to make an even better video.

The most rewarding of all, though, was seeing what the AISC community came up with. Take a look for yourself if you’d like. You’ll really enjoy the videos!

Leave a Reply

Your email address will not be published. Required fields are marked *