Research

My research is motivated by a key problem of AI which is making decisions in an interactive environment, besides perception and prediction. I am broadly interested in interactive machine learning (e.g. reinforcement learning and bandit) under the real-world constraints such as sample efficiency, safety, robustness and alignment.

Recently, I work on reinforcement learning methods in training foundation models (project: Amazon Bedrock) and foundation models for decision making.

Education and Experiences

• Research Scientist at AWS, Aug 2022 - Now
  
• Research Scientist at ByteDance, Aug 2021 - Aug 2022
  
• Ph.D. in Computer Science at Stanford, June 2021
  
• B.S. in Machine Intelligence at Peking University, June 2016
  

Preprints and Publications

• Budgeting Counterfactual for Offline RL
  Yao Liu, Pratik Chaudhari, Rasool Fakoor
  NeurIPS 2023 arXiv
  
• TD Convergence: An Optimization Perspective
  Kavosh Asadi, Shoham Sabach, Yao Liu, Omer Gottesman, Rasool Fakoor
  NeurIPS 2023 arXiv
  
• Provably Sample-Efficient RL with Side Information about Latent Dynamics
  Yao Liu, Dipendra Misra, Miro Dudík, Robert E. Schapire.
  NeurIPS 2022 arXiv
  
• Offline Policy Optimization with Eligible Actions
  Yao Liu, Yannis Flet-Berliac, Emma Brunskill.
  UAI 2022 arXiv
  
• Provably Good Batch Reinforcement Learning Without Great Exploration
  Yao Liu, Adith Swaminathan, Alekh Agarwal, Emma Brunskill.
  NeurIPS 2020 arXiv Code Talk
  
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
  Yao Liu, Pierre-Luc Bacon, Emma Brunskill
  ICML 2020 arXiv Talk
  
• Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions
  Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo Anthony Celi, Emma Brunskill, Finale Doshi-Velez
  ICML 2020 arXiv Code
  
• All-Action Policy Gradient Methods: A Numerical Integration Approach
  Benjamin Petit, Loren Amdahl-Culleton, Yao Liu, Jimmy Smith, Pierre-Luc Bacon arXiv
  
• Off-Policy Policy Gradient with State Distribution Correction
  Yao Liu, Adith Swaminathan, Alekh Agarwal, Emma Brunskill
  UAI 2019 (Oral) Supplemental arXiv
• Combining Parametric and Nonparametric Models for Off-Policy Evaluation
  Omer Gottesman, Yao Liu, Scott Sussex, Emma Brunskill, Finale Doshi-Velez
  ICML 2019 (Oral) arXiv Slides
• Representation Balancing MDPs for Off-Policy Policy Evaluation
  Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, Emma Brunskill.
  NeurIPS 2018 arXiv Code Talk
  
• When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithms
  Yao Liu, Emma Brunskill.
  EWRL 2018 arXiv
  
• Behaviour Policy Estimation in Off-Policy Evaluation: Calibration Matters
  Aniruddh Raghu, Omer Gottesman,Yao Liu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, Emma Brunskill
  ICML 2018 Workshops
  
• Switched Trajectories for Off-Policy Learning
  Scott Sussex , Omer Gottesman,Yao Liu, Susan Murphy, Emma Brunskill, Finale Doshi-Velez
  ICML 2018 Workshops
  
• Model Selection for Off-Policy Policy Evaluation
  Yao Liu, Philip S. Thomas, Emma Brunskill
  RLDM 2017, Extended Abstract
  
• PAC Continuous State Online Multitask Reinforcement Learning with Identification
  Yao Liu, Zhaohan Guo, Emma Brunskill
  AAMAS 2016 Supplemental
  
• Local Orthogonality Preserving Alignment for Nonlinear Dimensionality Reduction
  Tong Lin, Yao Liu, Bo Wang, Liwei Wang, Hongbin Zha
  Journal of Computer Science and Technology, 31(3): 512-524, 2016. Code
  

Teaching

CS234: Reinforcement Learning, Teaching Assistant, Winter 2019-2020.

CS229: Machine Learning, Teaching Assistant, Spring 2020-2021.

Professional Service

Journal Reviewing: JMLR, IEEE TPAMI, MLJ, AIJ, Biometrika

Conference Reviewing: NeurIPS (2019 - 2021), ICLR (2019 - 2021, 2023), ICML(2020, 2021), AISTATS (2020 - 2022), UAI (2020), AAAI (2022, 2023)