Homework 3

caution
  • You are expected to work individually.
  • Due: Monday March 24th at 11pm EST (Baltimore time).
  • This assignment is worth 100 points.
Objectives
  • Mutation Analysis & Testing
  • Mock Testing
  • GUI Testing
  • PIT, Mockito, & Selenium Tools
tip

You can get all the starter files for task 2 and task 3 of this homework by doing a git pull on the course public repo. The files can be found under hw/hw3.

Task 1 (23 points)#

Answer the following quesiotns:

  1. Why does it make sense to remove ineffective test cases (those that do not kill any mutant) from the test suite?

  2. The fundamental premise of mutation testing is stated as: “In practice, if the software contains a fault, there will usually be a set of mutants that can be killed only by a test case that also detects that fault.”

    • Give a brief argument in support of this premise.
    • Give a brief argument against this premise.

  3. Name two benefits and two downsides of Interaction-based testing (aka Behavior-based testing), especially in contrast with State-based testing.

  4. Write test cases to kill (strongly) mutant 1, 2, 3, and 4 in the following code:

int min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
🔺1 if (B > A) {
minVal = B;
🔺2 Bomb();
🔺3 minVal = A;
🔺4 minval = failOnZero(B);
}
return minVal;
}
info

A test case in this case is comprised of test input values and the corresponding expected value.

  1. Answer questions a through d for the mutant on line 5’ in the method sum().
    • a) If possible, find a test case that do not reach the mutant.
    • b) If possible, find a test case that satisfy reachability but not infection for the mutant.
    • c) If possible, find a test case that satisfy infection, but not propagation for the mutant.
    • d) If possible, find a test case that strongly kill the mutant.
/**
* Sum the values in an array
*
* @param x array to sum
*
* @return sum of values in x
* @throws NullPointerException if x is null
*/
1. public static int sum(int[] x) {
2. int s = 0;
3. for (int i=0; i < x.length; i++) {
4. s = s + x[i];
5’. // s = s - x[i]; //AOR
6. }
7. return s;
8. }

  1. We discussed testing a Graphical User Interface (GUI) but we did not mention any specific coverage criteria to make our GUI testing efforts (whether manual or using tools and frameworks) methodical. Suggest one or two coverage criteria for UI testing and give simple actual examples of them.

  2. All failures we detect during GUI testing are real and relevant because they are things that the end-user could potentially face too, but this is not necessarily the case when we do lower levels of testing, for instance, unit testing. In fact, some detected/reported failures during unit testing might be “false failures”. Explain using actual examples why this is indeed true.

Deliverables

The deliverables for this task is your answers to the questions above saved in a document named task1 e.g. task1.pdf.

Task 2: Mutation Analysis/Testing and PIT (30 points)#

You will be working with MathUtils.java for this task:

  1. Write as mnay test cases as you can think of (in Junit 5) for the MathUtils class based on specifications of each method.
info

Try to write as many different test cases as you can; think about all main/special/corner/edge/boundary etc. cases.

Store your test cases into MathUtilsTest.java.

  1. When doing (1), you should be able to find ambiguity, inconsistencies, mistakes, and/or lack of details in the specifications and/or the code. Fix the issues that you find and save your work in a separate class named MathUtilsFixed.java. Make sure to make a note of all the modifications you make in the form of comments in the new file.
  2. Manually generate one equivalent mutant for each method of MathUtils class and save them in a new file named MathUtilsEquivalent.java. If you think it is not possible to write an equivalent mutant for a method, provide an explanation why in MathUtilsEquivalent.java as a comment.
tip

Typically, mutation is just one small change to the original code. But, if it is not possible to produce an equivalent mutant by applying one mutation operator once, then feel free to use multiple mutations.

  1. Run PIT on MathUtilsFixed.java and your JUnit tests. What is line coverage and what is the mutation score of your test suite as reported by PIT?
info

Write line coverage and mutation percentage score numbers as comment lines on top of MathUtilsTest.java in this form:

/*
Line coverage before improvement: XX%
Mutation score before improvement: YY%
*/
  1. Next, reflect on those numbers by analyzing the PIT-generated report and try to improve them further. If needed, add more test cases to your JUnit file to increase the mutation score. If you add any new JUnit tests, make sure you leave a comment on top of it that reads: "added after Mutation analysis". Aim for 100% mutation score (killing all the mutants). If it is not possible to achieve that, explain why in the form comments on the top of the JUnit class file. Finally, generate a new PIT report using your improved MathUtilsTest.java and MathUtilsFixed.java and save it in html format.
Deliverables

Deliverables for this task are MathUtilsFixed.java, MathUtilsTest.java, MathUtilsEquivalent.java, and the (final) PIT report in html format saved in pitreport folder all saved under task2 folder.

Task 3: Mock Testing and Mockito(20 points)#

In this part, you will do mock testing on the code available on the public repo. In particular, we would like to test the Customer class (i.e., Customer is the class under test). Classes Account, Checking, Savings, Loan, and CreditReport are collaborators (i.e., dependencies) of the Customer class. Class Checking and Savings are subclasses of the Account class. Unfortunately, the real "Account" classes and the Loan class would make unrecoverable updates to the database and CreditReport needs fee payments to a credit reporting agency. For these reasons, we would like to use mocks instead of the real classes; the ones given to you are not the real ones. Your job is to write mock tests to check all the possible interactions between the Customer class and its collaborator classes. An interaction is when a method of some class calls a method of another class. In this case, we are only interested in interactions between Customer and the other classes. Note that for any possible interaction, you need to have at least one test case i.e. all interactions must be covered in your tests. To accomplish this, you are required to use Mockito. For consistency reasons in grading, you cannot use other mock testing tools.

order of interactions

If there are multiple interactions that are triggered together, you must verify the correct order of interactions in your mock tests.

Deliverables

The deliverables for this part are 1) a list of all possible interactions between Customer class with all account classes (Account, Savings and Checking), the Loan class, and the CreditReport class, and 2) a JUnit test class named CustomerMockTest.java that uses Mockito constructs where there is at least one interaction-based test case verifying each possible interaction. For (1), list all the interactions at the top of the JUnit test class as comments. Also, for each of the listed interactions, make sure to specify the condition(s) under which the interaction is triggerred. Finally, leave a comment before each test case to explain what interaction you are exactly testing in that particular test case. Please note in order to get full points here you need to cover all the possible interactions. Save the file under task3 folder.

Task 4: GUI Testing and Selenium (27 points)#

Using JUnit 5 and Selenium Webdriver, write separate JUit test methods to automate the followings:

Visit http://the-internet.herokuapp.com/:

  1. [test scenario 1]: on upload endpoint, assert that an attempt to upload while no file is chosen fails!
  2. [test scenario 2]: on hovers endpoint, assert that hovering on the leftmost box shows user1.
  3. [test scenario 3]: on checkboxes endpoint, assert that checkbox 1 is uncheked and checkbox 2 is selected.
  4. [test scenario 4]: on context_menu endpoint, assert that right-clicking in the box pops up a message box.
  5. [test scenario 5]: on broken_images endpoint, assert that there are two broken images.
  6. [test scenario 6]: on infinite_scroll endpoint, scroll down by 195.
  7. [test scenario 7]: on dropdown endpoint, select Option 2.
  8. [test scenario 8]: on key_presses endpoint, press the down key and assert that the page says You entered: DOWN.
  9. [test scenario 9]: on login endpoint (i.e., form authentication), try to login with empty user name and password and assert the page says Your username is invalid!
  10. [test scenario 10]: on login endpoint (i.e., form authentication), try to login with user name tomsmith and empty password and assert the page says Your password is invalid!
  11. [test scenario 11]: on login endpoint (i.e., form authentication), try to login with user name tomsmith and password SuperSecretPassword! and assert that: 1) the page redirects to secure endpoint, 2) the page says You logged into a secure area!, and 3) a button labeled Logout is both visible and enabled on the page. Click on the Logout button. Assert that you are back on login route and the page says You logged out of the secure area!
Deliverables

The deliverables for this part are your JUnit test class(es) containing the above test scenarios as a separate project. For each of the JUnit test methods you write, leave a short comment to indicate which of the above automations it corresponds to. Save the project folder as TheInternerGUITesting under task2 folder.

Selenium

Refer to exercise 10 on how to set up your Intellij and Gradle to utilize Selenium!

Submission#

Submission

Submissions should be made to Gradescope. Create a zip file named hw3.zip which contains all the deliverables for all the parts. For each task, create a separate folder named TaskX, where X is the part number, under which you put all the deliverables for that task. For each task, make sure to submit all the necessary deliverables including JUnit test classes (.java files NOT .class files), documents, reports etc.

danger

Important Note: If any of your .java files is not compilable/executable for any reason, you will receive a zero for that part. Thus, please make sure all the source files you hand in successfully compile and run!

Edit this page