Abstract

Programming-by-example (PBE) is the task of generating a program that is consistent with a set of user-provided input-output examples. Compared to generating code from natural language descriptions using LLMs, PBE is applicable to low-resource domains without a large number of comments to code pairs. A key challenge of PBE systems is ambiguity, where given a set of examples and multiple programs all consistent with these examples, the synthesizer must pick one program most likely intended by the user. I will present a body of work that resolves this ambiguity using pragmatic inference (Rational Speech Acts) from cognitive science, where the synthesizer models the user as a speaker who chooses examples informatively. Specifically, I will show (1) end users communicate more effectively with a pragmatic synthesizer, (2) pragmatic synthesis can be scaled to large neuro-symbolic domains using self-play, and (3) pragmatic reasoning has a theoretical connection to preference learning systems (e.g. RLHF).

About Evan Pu

Evan Pu did his PhD at MIT advised by Armando Solar-Lezama and Leslie Kaelbling. He is now a Senior Research Scientist at AutoDesk Research. In his own words: “I develop program synthesis algorithms that ground human instructions as codes, in a way a programmer would. A programmer does much more than generating Python from comments. Given an instruction (stated in language, test cases, or drawings), a human programmer can infer latent goals, generate code appropriate for the current situation, and interact to repair misunderstandings and overcome implementation difficulties. My work studies natural programs — instructions given to a person to carry out a task on a computer (e.g. modifying a 3D scene or coding). I curate datasets of natural programs, and build program synthesis systems inspired by cognitive science that grounds them as executable code, in a way a human would. I work on neuro-symbolic methods, computational pragmatics, and grounded semantics of natural language.”

Curious about our other news? Subscribe to our news feed, calendar, or newsletter, or follow us on social media.