Recommendation System Project — মেশিন লার্নিং

Hook — ‘তোমার জন্য’

Netflix watch time এর ~80% recommendation থেকে আসে। একটা সঠিক recommender মানেই কোটি টাকার engagement।

Problem Setup

Goal: user কে এমন item দেখাও যা সে likely click/buy/watch করবে।
Dataset: MovieLens 25M, Amazon Reviews, Last.fm।
Metric: Recall@K, NDCG@K, MAP, coverage, diversity।
Cold start — নতুন user/item এর জন্য আলাদা strategy।

Approach Stack

layers

1. Popularity baseline      → trending
2. Content-based            → tag/embedding similarity
3. Collaborative Filtering  → user-item matrix factorization (ALS, SVD)
4. Neural CF / Two-Tower    → deep embedding both sides
5. Hybrid + Re-ranking      → business rules, diversity, freshness

Matrix Factorization — Implicit ALS

als.py

import implicit, scipy.sparse as sp, pandas as pd

ratings = pd.read_csv("ratings.csv")
mat = sp.coo_matrix((ratings["rating"], (ratings["userId"], ratings["movieId"]))).tocsr()

model = implicit.als.AlternatingLeastSquares(factors=64, regularization=0.05, iterations=20)
model.fit(mat)

ids, scores = model.recommend(userid=10, user_items=mat[10], N=10)
print(ids)

Two-Tower Neural Model

two_tower.py

import torch, torch.nn as nn

class Tower(nn.Module):
    def __init__(self, n, d=64):
        super().__init__()
        self.emb = nn.Embedding(n, d)
        self.mlp = nn.Sequential(nn.Linear(d, 128), nn.ReLU(), nn.Linear(128, d))
    def forward(self, x): return self.mlp(self.emb(x))

user_tower, item_tower = Tower(n_users), Tower(n_items)

def score(u, i):
    return (user_tower(u) * item_tower(i)).sum(-1)

Training এ in-batch negative sampling ব্যবহার হয়। Inference এ user embedding compute → item index (FAISS) থেকে top-K nearest।

Two-Stage Serving

serving

Stage A — Candidate Generation
   Two-tower + ANN (FAISS / ScaNN)
   millions → 500 candidates  (~20ms)

Stage B — Ranking
   Gradient boosting / DLRM with rich features
   500 → 20 final              (~30ms)

Stage C — Business Re-rank
   diversity, freshness, sponsored, dedupe

Cold Start Strategy

New user → onboarding survey, demographic, top trending।
New item → content features (text, image embedding)।
Hybrid model — CF + content combined।
Bandit (Thompson Sampling) — explore vs exploit।

Evaluation

Offline: leave-one-out, time-based split, Recall@K, NDCG।
Online: A/B test — CTR, watch time, retention।
Guard metrics: diversity, coverage, fairness।

Deployment Notes

Item embedding nightly batch → FAISS index।
User embedding realtime (last actions থেকে)।
Feature store (Feast) → train/serve consistent।
Cache top-K per user (TTL 5 min)।

Summary

এক নজরে

Recommender = Candidate gen + Ranker + Re-rank। ALS baseline → Two-tower → DLRM ranker। Cold start, diversity, A/B — production এ অপরিহার্য।