The project:
After a machine learning (ML)-based system is deployed, monitoring its performance is important to ensure the safety and effectiveness of the algorithm over time. When an ML algorithm interacts with its environment, the algorithm can affect the data-generating mechanism and be a major source of bias when evaluating its standalone performance, an issue known as performativity. Although prior work has shown how to validate models in the presence of performativity using causal inference techniques, there has been little work on how to monitor models in the presence of performativity. The goal of this project is to bring together techniques from causal inference and statistical process control to develop a comprehensive framework for post-market monitoring of ML algorithms. This work builds on a number of our previous works, including this paper that was published at the Conference of Causal Learning and Reasoning (CLeaR) and was presented at the NeurIPS Regulatable AI workshop.
We are seeking a postdoctoral researcher to join our lab. The primary responsibilities are:
- Develop new statistical methods/frameworks for post-market monitoring of ML algorithms
- Implement a software package that can be readily used by ML developers, health AI deployment teams, and ML auditors/regulators
- Write, edit, and publish research manuscripts in collaboration with the team
Our team is highly collaborative and includes members with wide-ranging expertise:
- Jean Feng: PI of the lab. Primary advisor of postdoctoral researcher.
- Fan Xia: Co-advisor of postdoctoral researcher. Assistant Professor in the Department of Epidemiology and Biostatistics at UCSF. Research interests include causal inference, clinical trial design, and machine learning.
- Alexej Gossmann: Staff Fellow and mathematical statistician in the Division of Imaging, Diagnostics, and Software Reliability (CDRH/OSEL/DIDSR) at the FDA. Research interests include performance evaluation of AI/ML-enabled medical devices and software in medicine.
- And many others, including Berkman Sahiner, Gene Pennello, Adarsh Subbaswamy, Nicholas Petrick, Romain Pirracchio, and Mi-Ok Kim!
The position:
We are looking to hire a postdoctoral researcher to join the team. The position (100% funded) will be for two years. Salary and benefits are competitive.
Qualifications:
The post-doctoral researcher position requires at least a PhD degree in (bio)statistics, computer science, data science, or another relevant field. We are looking for someone who:
- has experience in at least one of these fields: sequential monitoring, machine learning, and causal inference
- has experience in methodological development and can perform independent research, with a strong and relevant publication record
- has strong software engineering background (e.g. python, git-based workflows, high-performance computing)
- is able to work collaboratively with a team
Applying
If you are interested, please submit the following materials to jean.feng@ucsf.edu:
- A cover letter
- A CV summarizing your education and work experience so far
- The names and email addresses of three references
- A code sample
- One representative publication
Screening of applicants will begin immediately and will continue as needed throughout the recruitment period.