Toward Rigorous Object-Code Coverage Criteria

Tae Joon Byun; Vaibhav B Sharma; Sanjai Rayadurgam; Stephen A McCamant; Mats Heimdahl

doi:10.1109/ISSRE.2017.33

Toward Rigorous Object-Code Coverage Criteria

Tae Joon Byun, Vaibhav B Sharma, Sanjai Rayadurgam, Stephen A McCamant, Mats Heimdahl

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Scopus citations

Abstract

Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers and the optimizations they perform can substantially change the structure of the generated code and the instructions used to represent branches. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantics of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC.

Original language	English (US)
Title of host publication	Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017
Publisher	IEEE Computer Society
Pages	328-338
Number of pages	11
ISBN (Electronic)	9781538609415
DOIs	https://doi.org/10.1109/ISSRE.2017.33
State	Published - Nov 14 2017
Event	28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017 - Toulouse, France Duration: Oct 23 2017 → Oct 26 2017

Publication series

Name	Proceedings - International Symposium on Software Reliability Engineering, ISSRE
Volume	2017-October
ISSN (Print)	1071-9458

Other

Other	28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017
Country/Territory	France
City	Toulouse
Period	10/23/17 → 10/26/17

Bibliographical note

Funding Information:
ACKNOWLEDGMENT This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The authors wish to thank Gregory Gay for his help in the tool development.

Funding Information:
This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444.

Publisher Copyright:
© 2017 IEEE.

Keywords

Coverage criteria
DO-178
OBC
Object-code testing
Structural testing

Access

10.1109/ISSRE.2017.33

OpenUrl availability

Full text

Cite this

Byun, T. J., Sharma, V. B., Rayadurgam, S., McCamant, S. A., & Heimdahl, M. (2017). Toward Rigorous Object-Code Coverage Criteria. In Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017 (pp. 328-338). Article 8109098 (Proceedings - International Symposium on Software Reliability Engineering, ISSRE; Vol. 2017-October). IEEE Computer Society. https://doi.org/10.1109/ISSRE.2017.33

Toward Rigorous Object-Code Coverage Criteria. / Byun, Tae Joon; Sharma, Vaibhav B; Rayadurgam, Sanjai et al.
Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017. IEEE Computer Society, 2017. p. 328-338 8109098 (Proceedings - International Symposium on Software Reliability Engineering, ISSRE; Vol. 2017-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Byun, TJ, Sharma, VB, Rayadurgam, S , McCamant, SA & Heimdahl, M 2017, Toward Rigorous Object-Code Coverage Criteria. in Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017., 8109098, Proceedings - International Symposium on Software Reliability Engineering, ISSRE, vol. 2017-October, IEEE Computer Society, pp. 328-338, 28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017, Toulouse, France, 10/23/17. https://doi.org/10.1109/ISSRE.2017.33

Byun TJ, Sharma VB, Rayadurgam S , McCamant SA , Heimdahl M. Toward Rigorous Object-Code Coverage Criteria. In Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017. IEEE Computer Society. 2017. p. 328-338. 8109098. (Proceedings - International Symposium on Software Reliability Engineering, ISSRE). doi: 10.1109/ISSRE.2017.33

@inproceedings{0bf76566962f483bbb1642b58216b4b4,

title = "Toward Rigorous Object-Code Coverage Criteria",

abstract = "Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers and the optimizations they perform can substantially change the structure of the generated code and the instructions used to represent branches. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantics of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC.",

keywords = "Coverage criteria, DO-178, OBC, Object-code testing, Structural testing",

author = "Byun, {Tae Joon} and Sharma, {Vaibhav B} and Sanjai Rayadurgam and McCamant, {Stephen A} and Mats Heimdahl",

note = "Funding Information: ACKNOWLEDGMENT This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The authors wish to thank Gregory Gay for his help in the tool development. Funding Information: This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444. Publisher Copyright: {\textcopyright} 2017 IEEE.; 28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017 ; Conference date: 23-10-2017 Through 26-10-2017",

year = "2017",

month = nov,

day = "14",

doi = "10.1109/ISSRE.2017.33",

language = "English (US)",

series = "Proceedings - International Symposium on Software Reliability Engineering, ISSRE",

publisher = "IEEE Computer Society",

pages = "328--338",

booktitle = "Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017",

}

TY - GEN

T1 - Toward Rigorous Object-Code Coverage Criteria

AU - Byun, Tae Joon

AU - Sharma, Vaibhav B

AU - Rayadurgam, Sanjai

AU - McCamant, Stephen A

AU - Heimdahl, Mats

N1 - Funding Information: ACKNOWLEDGMENT This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The authors wish to thank Gregory Gay for his help in the tool development. Funding Information: This material is based upon work supported by the National Science Foundation under Grant Numbers 1563920 and 1514444. Publisher Copyright: © 2017 IEEE.

PY - 2017/11/14

Y1 - 2017/11/14

N2 - Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers and the optimizations they perform can substantially change the structure of the generated code and the instructions used to represent branches. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantics of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC.

AB - Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers and the optimizations they perform can substantially change the structure of the generated code and the instructions used to represent branches. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantics of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC.

KW - Coverage criteria

KW - DO-178

KW - OBC

KW - Object-code testing

KW - Structural testing

UR - http://www.scopus.com/inward/record.url?scp=85040776988&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040776988&partnerID=8YFLogxK

U2 - 10.1109/ISSRE.2017.33

DO - 10.1109/ISSRE.2017.33

M3 - Conference contribution

AN - SCOPUS:85040776988

T3 - Proceedings - International Symposium on Software Reliability Engineering, ISSRE

SP - 328

EP - 338

BT - Proceedings - 2017 IEEE 28th International Symposium on Software Reliability Engineering, ISSRE 2017

PB - IEEE Computer Society

T2 - 28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017

Y2 - 23 October 2017 through 26 October 2017

ER -

Toward Rigorous Object-Code Coverage Criteria

Abstract

Publication series

Other

Bibliographical note

Keywords

Access

OpenUrl availability

Other files and links

Fingerprint

Cite this