Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning
The blog explores the challenges and advancements in post-training generative recommender systems, introducing Advantage-Weighted Supervised Fine-tuning (A-SFT) to improve recommendation quality by addressing issues with noisy reward models and lack of counterfactual feedback, outperforming traditional methods in offline evaluations.
MAIN POINTS
- Generative recommenders model user behavior as sequential tasks, inspired by transformer architectures.
- Challenges include noisy reward models, lack of counterfactual data, and unknown logged policies.
- A-SFT combines supervised fine-tuning with advantage reweighting to optimize post-training.
- Offline evaluations show A-SFT outperforms traditional methods in recommendation metrics.
TAKEAWAYS
- Generative recommenders benefit from user feedback beyond observed behavior to enhance recommendations.
- A-SFT effectively handles high variance in reward models by leveraging directional signals.
- The method does not rely on Inverse Propensity Scoring, reducing high-variance estimate risks.
- A-SFT achieves better alignment between pre-trained models and reward models, improving recommendation quality.