GPT-3-Powered Type Error Debugging: Investigating the Use of Large Language Models for Code RepairResearch Paper
Type systems are responsible for assigning types to terms in programs. That way, they enforce the actions that can be taken and can, consequently, detect type errors during compilation. However, while they are able to flag the existence of an error, they often fail to pinpoint its cause or provide a helpful error message. Thus, without adequate support, debugging this kind of errors can take a considerable amount of effort. Recently, neural network models have been developed that are able to understand programming languages and perform several downstream tasks. We argue that type error debugging can be enhanced by taking advantage of this deeper understanding of the language’s structure. In this paper, we present a technique that leverages GPT-3’s capabilities to automatically fix type errors in \textit{OCaml} programs. We perform multiple source code analysis tasks to produce useful prompts that are then provided to GPT-3 to generate potential patches. Our publicly available tool, \textsc{Mentat}, supports multiple modes and was validated on an existing public dataset with thousands of \textit{OCaml} programs. We automatically validate successful repairs by using Quickcheck to verify which generated patches produce the same output as the user-inteded fixed version, achieving a $39%$ repair rate. In a comparative study, \textsc{Mentat} outperformed two other techniques in automatically fixing ill-typed \textit{OCaml} programs.
Mon 23 OctDisplayed time zone: Lisbon change
16:00 - 17:30 | Inference and automationSLE at Room II Chair(s): Adrian Johnstone Royal Holloway University of London, UK | ||
16:00 30mTalk | Automated extraction of grammar optimization rule configurations in a metamodel-grammar co-evolution scenarioResearch Paper SLE Weixing Zhang Chalmers | University of Gothenburg, Regina Hebig Chalmers University of Technology & University of Gothenburg, Daniel Strüber Chalmers | University of Gothenburg / Radboud University, Jan-Philipp Steghöfer XITASO GmbH IT & Software Solutions DOI Pre-print | ||
16:30 30mTalk | Reuse and Automated Integration of Recommenders for Modelling LanguagesResearch Paper SLE Lissette Almonte Universidad Autónoma de Madrid, Antonio Garmendia Universidad Autónoma de Madrid, Esther Guerra Universidad Autónoma de Madrid, Juan de Lara Autonomous University of Madrid DOI Pre-print | ||
17:00 30mTalk | GPT-3-Powered Type Error Debugging: Investigating the Use of Large Language Models for Code RepairResearch Paper SLE Francisco Ribeiro HASLab/INESC TEC & Universidade do Minho, José Nuno Macedo University of Minho, Kanae Tsushima National Institute of Informatics, Japan, Rui Abreu Faculty of Engineering, University of Porto, João Saraiva HASLab/INESC TEC, University of Minho DOI |