USI - Email

Archive / INF Seminars / INF_2025_04_17_NargizHumbatova

USI - Email

	Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

Chair: Valerie Burgener

Thursday

17.04

USI Campus EST, Room D0.02

16:30 - 17:30


	Nargiz Humbatova Università della Svizzera italiana

Abstract: As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. We performed a manual analysis of 490 faults from five different benchmarks and identified that 314 of them are eligible for further analysis. Our investigation focused specifically on how well the faults correspond to the sources they were extracted from, which fault types are represented, and whether the faults are reproducible. Our findings indicate that only 18.5% of the faults satisfy realism conditions. Our attempts to reproduce these faults were successful only in 52% of cases.

Biography: Nargiz is a postdoctoral assistant in the TAU group at USI, working with Professor Paolo Tonella. She completed her PhD also at USI as part of the ERC project PRECRIME, under the supervision of Professor Paolo Tonella and Dr. Gunel Jahangirova. Prior to that, she earned her MSc from the University of Bristol and her BSc from Moscow State University.

********************************

In February 2019, the Software Institute started its SI Seminar Series. Every Thursday afternoon, a researcher of the Institute will publicly give a short talk on a software engineering argument of their choice. Examples include, but are not limited to novel interesting papers, seminal papers, personal research overview, discussion of preliminary research ideas, tutorials, and small experiments.

On our YouTube playlist you can watch some of the past seminars. On the SI website you can find more details on the next seminar, the upcoming seminars, and an archive of the past speakers