We measured the statement coverage of Javascript semantics against ECMAScript Conformance Test Suite.
Using K's --coverage option, we can measure the coverage of a given test, w.r.t. the standard.
$ krun --coverage test.js
In the coverage result, each semantic rule is annotated with a number
of how many times it was executed by the test suite. The number 0 means that
the corresponding semantic rule is not covered by any test.
This way we found that there are exactly 17 semantic rules in the core semantics which are not covered by the test suite, each corresponding to the language standard as shown in the following:
We succeeded to manually write test programs that hit 11 out of 17 behaviors: (Each link shows the corresponding test program.)
- [8.7.1 GetValue (V) - [[Get]], Step 6 ](01 - 8.7.1 GetValue (V) - [[Get]], Step 6.js)
- [8.7.2 PutValue (V, W) - [[Put]], Step 2.a ](02 - 8.7.2 PutValue (V, W) - [[Put]], Step 2.a.js)
- [8.7.2 PutValue (V, W) - [[Put]], Step 2.b ](03 - 8.7.2 PutValue (V, W) - [[Put]], Step 2.b.js)
- [8.7.2 PutValue (V, W) - [[Put]], Step 6.a & 6.b ](06 - 8.7.2 PutValue (V, W) - [[Put]], Step 6.a-b.js)
- [8.7.2 PutValue (V, W) - [[Put]], Step 7.a ](07 - 8.7.2 PutValue (V, W) - [[Put]], Step 7.a.js)
- [8.12.4 [[CanPut]] (P) - Step 8.a ](08 - 8.12.4 [[CanPut]] (P) - Step 8.a.js)
- [10.2.1.1.3 SetMutableBinding (N,V,S) - Step 4 ](09 - 10.2.1.1.3 SetMutableBinding (N,V,S) - Step 4.js)
- [10.2.1.1.5 DeleteBinding (N) - Step 4 & 5 ](12 - 10.2.1.1.5 DeleteBinding (N) - Step 4-5.js)
- [10.5 Declaration Binding Instantiation - Step 5.e.iii.1 ](14 - 10.5 Declaration Binding Instantiation - Step 5.e.iii.1.js)
- [10.5 Declaration Binding Instantiation - Step 5.e.iv, 1st condition is true ](15 - 10.5 Declaration Binding Instantiation - Step 5.e.iv, 1st.js)
- [10.5 Declaration Binding Instantiation - Step 5.e.iv, 2nd condition is true ](16 - 10.5 Declaration Binding Instantiation - Step 5.e.iv, 2nd.js)
Moreover, the remaining 6 semantic behaviors are infeasible, that is, they represent flaws in the language standard itself: (Each link presents a brief explanation of why it is infeasible.)
- [8.7.2 PutValue (V, W) - [[Put]], Step 4.a ](04 - 8.7.2 PutValue (V, W) - [[Put]], Step 4.a.js)
- [8.7.2 PutValue (V, W) - [[Put]], Step 4.b ](05 - 8.7.2 PutValue (V, W) - [[Put]], Step 4.b.js)
- [10.2.1.1.4 GetBindingValue(N,S) - Step 3.a ](10 - 10.2.1.1.4 GetBindingValue(N,S) - Step 3.a.js)
- [10.2.1.1.5 DeleteBinding (N) - Step 2 ](11 - 10.2.1.1.5 DeleteBinding (N) - Step 2.js)
- [10.2.1.2.4 GetBindingValue(N,S) - Step 4.a ](13 - 10.2.1.2.4 GetBindingValue(N,S) - Step 4.a.js)
- [10.6 Arguments Object - [[DefineOwnProperty]], Step 4.a, else-branch ](17 - 10.6 Arguments Object - [[DefineOwnProperty]], Step 4.a, else.js)