USI - Email

Archive / INF Seminars / INF_2022_10_06_Nargiz_Humbatova

USI - Email

	DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults

In February 2019, the Software Institute started its SI Seminar Series. Every Thursday afternoon, a researcher of the Institute will publicly give a short talk on a software engineering argument of their choice. Examples include, but are not limited to, novel interesting papers, seminal papers, personal research overview, discussion of preliminary research ideas, tutorials, and small experiments.

Thursday

06.10

USI Campus EST, room D0.03, Sector D

16:30 - 17:30


	Nargiz Humbatova Software Institute, Switzerland

news_img

Abstract:
Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Traditional test adequacy criteria are not effective with DL models, whose behaviour is determined by many factors such as training data, model architecture, a set of various hyperparameters, and only marginally by the source code. Hence, researchers have proposed several novel test adequacy criteria tailored for DL systems, including mutation adequacy (mutation testing). However, the newly proposed mutation testing approaches were mainly based on some random changes to the structure or the weights of an already trained DL model. In one of our recent works, we investigated the possibility to simulate the effects of real faults by means of DL-specific mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Finally, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.

Biography:
Nargiz is a fourth-year PhD student at TAU group, working on the PRECRIME project under the supervision of Professor Paolo Tonella and Assistant Professor Gunel Jahangirova. She obtained her BSc in Mathematics from Moscow State University in 2012. Then she spent almost five years working as a software developer in a bank in Baku, Azerbaijan. Before joining USI, she got an MSc degree in Advanced Computing from the University of Bristol. Currently, her research interests are focused on the topic of Mutation Testing of Deep Learning Systems, automated repair and fault localisation.