Bay Area Tech Economics Seminar with Nathan Kallus, Cornell/ Netflix

Speaker: Nathan Kallus, Associate Professor, Cornell University; Research Director of Machine Learning & Inference Research, Netflix

Presentation Title: Learning about Tradeoffs and Proxies from Historical A/B Tests

Presentation Abstract: Experimentation on digital platforms often faces a dilemma: we want to experiment rapidly at scale but we also want to make decisions based on long-term metrics that are both insensitive and take too long to observe. Usually one resorts to looking at short-term outcomes like engagement. However, with increasingly complex platforms, engagement can have many modalities (e.g., video vs games) and qualities (e.g., total time vs title completions). So a key question is how these different objectives tradeoff and how to combine them into the best proxy metric for one's north star. There are many pitfalls in doing this using observational data on users. We will discuss how to use meta-analysis of past experiments to answer this question in ways that are robust to confounding between short- and long-term and to unmediated effects and the impact of doing this at Netflix.

Speaker Bio: Nathan Kallus is an Assistant Professor in the School of Operations Research and Information Engineering and Cornell Tech at Cornell University. His research focuses on personalization, optimization under uncertainty, causal inference, sequential decision-making, robust inference, and algorithmic fairness. He holds a PhD in Operations Research from MIT and a BA in Mathematics and BS in Computer Science from UC Berkeley.

Before joining Cornell, Kallus was a Visiting Scholar at USC and a Postdoctoral Associate at MIT. He also serves as the Research Director of Machine Learning & Inference Research at Netflix, where he leads efforts in advancing recommendation systems and optimizing personalized content delivery.

Logistics details for each session will be provided to registered participants.

to be included on the mailing list and learn about future events.

For additional information and abstracts from past talks, .

Also check out other Workshops in Stanford, Nonprofit events in Stanford.