Unit Testing Overview

Basics of Python & Java

Python Unit Tests

Java Unit Tests

HTML/CSS/JS Unit Tests

JavaScript Unit Tests

Jupyter Notebook Tests

Python Unit Test

The basic building blocks of unit testing are test cases — single scenarios that must be set up and checked for correctness.

The simplest TestCase subclass will simply implement a test method (i.e. a method whose name starts with test) in order to perform specific testing code:

def test_default_case(self): 
        # Your test case logic here (replace the example assertion below) 
        # You may also rename this to any function in the form of 'test_your_test_name(self):' 
        self.assertTrue(True)

Note that in order to test something, we use one of the assert*() methods provided by the TestCase base class. If the test fails, an exception will be raised with an explanatory message, and unittest will identify the test case as a failure. Any other exceptions will be treated as errors.

Tests can be numerous, and their set-up can be repetitive. Luckily, we can factor out set-up code by implementing a method called setUp(), which the testing framework will automatically call for every single test we run:

def setUp(self): 
        # Setup code here (if required, replace the 'pass') 
        pass

Similarly, we can provide a tearDown() method that tidies up after the test method has been run:

def tearDown(self): 
        # Teardown code here (if required, replace the 'pass') 
        pass

If setUp() succeeded, tearDown() will be run whether the test method succeeded or not.

Here are some JUnit examples using similar setups as those above.

@Test public void testDefaultCase() { 
        // You may rename this method to better suit the purpose of your test case 
        // Your test case logic here 
}

@Before public void setUp() { 
        // Setup code here (if required) 
}

@After public void tearDown() { 
        // Teardown code here (if required) 
}

Here are various assertion methods that can be used when building a unit test.

Full documentation - https://docs.python.org/3/library/unittest.html

Why use assertListEqual() instead of just assertEqual()? Well, they are designed to show the difference between the two values on failure. Learners will receive more accurate feedback when the proper assertion is used.

Feedback can be added to assertion failures to alert learners what they might have done wrong. To do so, just simply add an additional argument as a string containing the message to display.

assertEqual(learner_output, real_output, "Looks like your code is not outputing the proper value.")

Click here for more info on skipping test and expected failures.

Full documentation - ​https://junit.org/junit4/javadoc/4.12/org/junit/Assert.html

Add feedback to assertion failures to alert learners what they might have done wrong. Simply add a string containing the message to display as the first argument in the method call.

assertEquals("Looks like your code is not outputing the proper value.", realOutput, learnerOutput)

Python Unit Tests

To check a variable value you need to import the learner's code and use dot notation to get the value.

For example, maybe you ask learners to assign a variable named num1 the value of 5.

Filename: example.py

num1 = 5

Grading Tests:

import unittest 
class CodingRoomsUnitTests(unittest.TestCase):
    def test_default_case(self): 
        import example 
        self.assertTrue(example.num1 == 5)

You can do anything you need to with a learner's variable value. For example, we can see if a learner created a list with at least 3 items using the len() function.

Filename: example.py

my_list = ["first thing", "second thing", "third thing"]

Grading Tests:

import unittest

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import example
        self.assertTrue( len(example.my_list) >= 3 )

This is an example of how you can test the output stream of learner code.

This test makes sure the learner's code prints "Hello World". It assumes the learner's file name is example.py. Replace the import with whatever your learner's file name is.

Grading Tests:

import unittest
import sys, io

stdout = sys.stdout

class CodingRoomsUnitTests(unittest.TestCase):

    def setUp(self):
        global stdout
        stdout = sys.stdout
        sys.stdout = io.StringIO()

    def tearDown(self):
        global stdout
        sys.stdout = stdout
        
    def test_default_case(self):
        import example
        
        output = sys.stdout.getvalue().strip("\n")

        answer = "Hello World".casefold().replace(" ", "")
        learner = output.casefold().replace(" ", "")
        
        self.assertEqual(answer, learner)

Filename: example.py

print("Hello World")

This is an example of how you can mock input values and test the output stream of learner code.

The learner must use the input() function to prompt the user for their first name, then again for their last name. Then, print Hello, firstname lastname! including the person's first and last name.

Filename: example.py

What is your first name? Jane 
What is your last name? Doe 
Hello, Jane Doe!

This code converts the output to all lowercase and removes spaces to compare against the correct value. Replace the import with whatever your learner's file name is.

import unittest
import sys, io
import importlib

stdout = sys.stdout
stdin = sys.stdin
learner_code = ""

class CodingRoomsUnitTests(unittest.TestCase):

    def setUp(self):
        global stdout
        global stdin
        stdout = sys.stdout
        stdin = sys.stdin

    def tearDown(self):
        global stdout
        global stdin
        sys.stdin = stdin
        sys.stdout = stdout

    def test_case_1(self):
        global learner_code
        sys.stdout = io.StringIO()
        sys.stdin = io.StringIO("Jane\nDoe")

        learner_code = importlib.import_module('example')

        output = sys.stdout.getvalue().strip("\n")
        
        self.assertTrue(output.casefold().replace(" ", "").endswith("Hello, Jane Doe!".casefold().replace(" ", "")))

    def test_case_2(self):
        global learner_code
        sys.stdout = io.StringIO()
        sys.stdin = io.StringIO("Joe\nMazzone")

        importlib.reload(learner_code)

        output = sys.stdout.getvalue().strip("\n")
        
        self.assertTrue(output.casefold().replace(" ", "").endswith("Hello, Joe Mazzone!".casefold().replace(" ", "")))

The io.StringIO() for stdin replaces the program’s input stream. Each input must be separated by a newline character -\n.

You will also notice that the assertion doesn't compare the output directly with what is expected. Instead, it sees if the output stream .endswith() the expected output.

This is because the input() function's prompt is included in the output stream, so we only want to check the last thing printed. If you want to include the prompts to ensure learners properly included them, you can use assertEqual() and compare the entire output stream

Filename: example.py

fname = input("What is your first name?") 
lname = input("What is your last name?") 

print("Hello, " + fname + " " + lname + "!")

This is an example of how you can test the output stream of a learner-defined function.

This test makes sure the learner's function named print_hello() prints "Hello World". It assumes the learner's file name is example.py. Replace the import with whatever your learner's file name is.

import unittest
import sys, io

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        stdout = sys.stdout
        sys.stdout = io.StringIO()
        import example
        example.print_hello()
        output = sys.stdout.getvalue().strip("\n")
        sys.stdout = stdout
        answer = "Hello World".casefold().replace(" ", "")
        learner = output.casefold().replace(" ", "")
        self.assertEqual(answer, learner) 

Filename: example.py

def print_hello():
    print("Hello World")

Each example assumes the learner’s file name is example#.py replacing # with the actual example number. You can call the functions by importing the learner file name of your choice.

Learners write a function named add_five() that takes an integer as a parameter and returns 5 plus the parameter value.

import unittest
import sys, io

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import example1
        self.assertEqual(10, example1.add_five(5))
        self.assertEqual(5, example1.add_five(0))
        self.assertEqual(11, example1.add_five(6))

Filename: example1.py

def add_five(number):
    number += 5
    return number

Learners write a function that takes 5 values as parameters and returns the average value of the 5 values. The function must be named average() and it must have 5 parameters.

Example: Calling average(1, 5, 7, 4, 10) would return 5.4.

import unittest
import sys, io

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import example2
        self.assertAlmostEqual(5.4, example2.average(1, 5, 7, 4, 10), delta=0.001)
        self.assertAlmostEqual(9.6, example2.average(5, 9, 24, 6, 4), delta=0.001)

Filename: example2.py

def average(a, b, c, d, e):
    sum = a + b + c + d + e
    return sum / 5

Learners write a function that takes a parameter. If the String parameter has a double letter (i.e. contains the same letter twice in a row) then it should return true. Otherwise, it should return false.

This function must be named has_repeat() and have a parameter. This function must return a boolean.

import unittest
import sys, io

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import example3
        self.assertTrue(example3.has_repeat("mississippi"))
        self.assertFalse(example3.has_repeat("capsized"))
        self.assertTrue(example3.has_repeat("mazzone"))
        self.assertFalse(example3.has_repeat("this"))

Filename: example3.py

def has_repeat(word):
    count = 0
    for letter in word:
        if letter == word[count-1]:
            return True
        count += 1
    return False

Testing code that includes random numbers can be hard. Luckily, the random module comes with the seed() function.

The seed() function allows you to initialize the random number generator with a specific value , which forces it to produce the same values each time.

In this example, learners are supposed to create a function named make_random_numbers() which returns a list of 10 random numbers between 20 and 99 inclusive.

import unittest
import random

class CodingRoomsUnitTests(unittest.TestCase):

    def test_case1(self):
        import example
        random.seed(10)
        self.assertListEqual([93, 24, 74, 81, 93, 21, 46, 79, 82, 55], example.make_random_numbers())

    def test_case2(self):
        import example
        random.seed(55)
        self.assertListEqual([31, 45, 39, 58, 30, 43, 58, 31, 65, 80], example.make_random_numbers() )
    
    def test_case3(self):
        import example
        random.seed(2)
        self.assertListEqual([27, 31, 30, 66, 41, 59, 52, 97, 47, 97], example.make_random_numbers())

Notice that random.seed() is called, setting the randomly generated numbers to predetermined values, before you check if the functions return value equals the expected value.

Filename: example.py

import random

def make_random_numbers():
    numbers_list = []
    for num in range(10):
        numbers_list.append(random.randint(20,99))
    return numbers_list

Even better, we can randomly generate the seed and make the test random each time!

This example expects the learner to use the randint() function to generate and print a random number between 0 and 500.

import unittest
import sys, io
import random

stdout = sys.stdout

class CodingRoomsUnitTests(unittest.TestCase):

    def setUp(self):
        global stdout
        stdout = sys.stdout
        sys.stdout = io.StringIO()

    def tearDown(self):
        global stdout
        sys.stdout = stdout
        
    def test_default_case(self):
        s = random.randint(1, 55)
        
        random.seed(s)
        import example
        output = sys.stdout.getvalue()

        random.seed(s)
        num = random.randint(0, 500)

        answer = str(num)
        learner = output
        
        self.assertEqual(answer, learner)

Filename: example.py

import random

random_num = random.randint(0,500)

print(random_num)

Here is another where the learner needed to print a random item from a list they created named learner_list.

import unittest
import sys, io
import random

stdout = sys.stdout

class CodingRoomsUnitTests(unittest.TestCase):

    def setUp(self):
        global stdout
        stdout = sys.stdout
        sys.stdout = io.StringIO()

    def tearDown(self):
        global stdout
        sys.stdout = stdout
        
    def test_default_case(self):
        s = random.randint(1, 55)
        
        random.seed(s)
        import example
        
        output = sys.stdout.getvalue().strip("\n")
        learner = output

        random.seed(s)
        answer = random.choice(example.learner_list)
        
        self.assertEqual(answer, learner)/pre>

Filename: example.py

import random

learner_list = ["first thing", "second thing", "third thing", "fourth thing"]

random_thing = random.choice(learner_list)

print(random_thing)

check if learner used _______ in their code.

Often you just want to make sure learners successfully use a particular function/command/structure. The simplest way I have found to do this is just to parse the learner's code file and make sure what you are looking for is there.

For example, maybe you want to make sure the learner used the input function 4 times. We can parse the learner's code file and count how many times "input(" is present in their code.

This test assumes the learner's file name is example.py.

import unittest

count = 0

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        global count
        f = open("example.py", "r")
        lines = f.readlines()
        f.close()
        
        for line in lines:
            line = line.strip().split()
            for words in line:
                if words.find("input(") != -1:
                    count += 1
                    
        self.assertTrue(count == 4)

Notice that the test asserts that the count is equal to 4. Of course, you can change this condition to anything you want. For example, if you want to make sure they have at least 1 input() you could use count >= 1.

Filename: example.py

name = input("What is your name?")
age = input("What is your age?")
hair = input("What color is your hair?")
place = input("Where do you live?")

Sometimes if you are checking for a specific character/symbol it is better to use the .count() method so you do not miss an instance of the character/symbol.

This example checks that the learner used the + addition at least 2 times.

import unittest

count = 0

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        global count
        f = open("example.py", "r")
        lines = f.readlines()
        f.close()
        
        for line in lines:
            line = line.strip().split()
            for words in line:
                if words.find("+") != -1:
                    count += words.count("+")

        self.assertTrue(count >= 2)

Since spaces can be used between arguments, to check if a function is called with exact arguments, you want to check each line of the learner's code and remove all spaces. Then find the function call you are looking for.

This example checks that there is at least 1 call of randint(501,1000) in the learner's code.

import unittest

count = 0

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        global count
        f = open("example.py", "r")
        lines = f.readlines()
        f.close()
        
        for line in lines:
            line = line.strip().replace(" ", "")
            if line.find("randint(501,1000)") != -1:
                count += 1
                
        self.assertTrue(count >= 1)

Java Unit Tests

Checking variable values can be a bit tricky in Java. For example, it can be hard to check the following variable values.

class Example 
{
    public static void main(String[] args) 
    {
        String firstName = "Bob";
        String lastName = "Jones";
        System.out.println(firstName);
        System.out.println(lastName);
    }
}

This is because firstName and lastName are local to the main() method and created when it is executed and destroyed when it is done.

We can prevent this by declaring these variables as static outside the main() method.

class Example
{
    static String firstName;
    static String lastName;
 
    public static void main(String[] args)
    {
        firstName = "Bob";
        lastName = "Jones";
        System.out.println(firstName);
        System.out.println(lastName);
    }
}

We typically do not teach learners to declare variables this way (most curriculums do not) so you may need to tweak your curriculum to auto-grade early assignments that teacher variable assignment and console output.

The grading test involves calling the main method (where you expect the variables to be assigned values and checking that each variable equals the expected value.

import org.junit.Before;
import org.junit.After;
import org.junit.Test;
import static org.junit.Assert.*;

public class CodingRoomsUnitTests {

    @Test
    public void testDefaultCase() {
        Example.main(null);
        assertEquals(Example.firstName, "Bob");
        assertEquals(Example.lastName, "Jones");
    }
}

This is an example of how you can test the output stream of learner code.

This test makes sure the learner's code prints "Hello World". It assumes the learner's file name is Example.java. Replace the Example.main(null) call with whatever you learner's file name is.

This code converts the output to all lowercase and removes spaces to compare against the correct value.

import org.junit.Before;
import org.junit.After;
import org.junit.Test;
import static org.junit.Assert.*;
import java.io.*;


public class CodingRoomsUnitTests {
    
    private final PrintStream standardOut = System.out;
    private final ByteArrayOutputStream outputStreamCaptor = new ByteArrayOutputStream();

    @Before
    public void setUp() {
        System.setOut(new PrintStream(outputStreamCaptor));
    }

    @After
    public void tearDown() {
        System.setOut(standardOut);
    }

    @Test
    public void testDefaultCase() {
        Example.main(null);
        assertEquals("Hello World".toLowerCase().trim().replace(" ", ""), outputStreamCaptor.toString().toLowerCase().trim().replace(" ", ""));
    }    
}

example.java

public class Example
{
    public static void main(String[] args)
    {
        System.out.println("Hello World");
    }
}

This is an example of how you can mock input values and test the output stream of learner code.

The learner must use the Scanner'snextLine() function to prompt the user for their first name, the again for their last name. Then, print Hello, firstname lastname! including the person's first and last name.

What is your first name? 
Jane
What is your last name? 
Doe
Hello, Jane Doe!

It assumes the learner's file name is Example.java. This can be easily replaced with whatever your learner's file name is. This code converts the output to all lowercase and removes spaces to compare against the correct value.

import org.junit.Before;
import org.junit.After;
import org.junit.Test;
import static org.junit.Assert.*;
import java.io.*;

public class CodingRoomsUnitTests {
    
    private final PrintStream standardOut = System.out;
    private final ByteArrayOutputStream outputStreamCaptor = new ByteArrayOutputStream();

    @Before
    public void setUp() {
        System.setOut(new PrintStream(outputStreamCaptor));
    }

    @After
    public void tearDown() {
        System.setOut(standardOut);
    }

    @Test
    public void testCase1() {
        String input = "Joe\nMazzone";
        InputStream in = new ByteArrayInputStream(input.getBytes());
        System.setIn(in);
        Example.main(null);
        assertTrue(outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "").endsWith("Hello Joe Mazzone!".toLowerCase().trim().replace(" ", "")));
    }

    @Test
    public void testCase2() {
        String input = "Clark\nKent";
        InputStream in = new ByteArrayInputStream(input.getBytes());
        System.setIn(in);
        Example.main(null);
        assertTrue(outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "").endsWith("Hello Clark Kent!".toLowerCase().trim().replace(" ", "")));
    }

    @Test
    public void testCase3() {
        String input = "Bruce\nWayne";
        InputStream in = new ByteArrayInputStream(input.getBytes());
        System.setIn(in);
        Example.main(null);
        assertTrue(outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "").endsWith("Hello Bruce Wayne!".toLowerCase().trim().replace(" ", "")));
    }
}

The input String replaces the programs input stream. Each input must be separated by a newline character -\n.

You will also notice that the assertion doesn't compare the output directly with what is expected. Instead it sees if the output stream .endsWith() the expected output. This is because the we print other values as prompts for the Scanner input, so we only want to check the last thing printed.

If you want to include the prompts to ensure learners properly included them, you can use assertEqual() and and compare the entire output stream.

Example.java

import java.util.Scanner;

public class Example
{
    public static void main(String[] args)
    {
        Scanner scan = new Scanner(System.in);
        
        System.out.println("What is your first name?");
        String fName = scan.nextLine();
        
        System.out.println("What is your last name?");
        String lName = scan.nextLine();
        
        System.out.println("Hello, " + fName + " " + lName + "!");
    }
}

Each example assumes the learner’s file name is Example#.java replacing # with the actual example number. You can call the methods based on the learner filename of your choice.

Learners write an int method named addFive(int number) that takes an integer as a parameter and returns an int equal to 5 plus the parameter value.

import org.junit.Test;
import static org.junit.Assert.*;

public class CodingRoomsUnitTests {
    @Test
    public void testDefaultCase() {
        // You may rename this method to better suit the purpose of your test case
        // Your test case logic here
        assertEquals(10, Example1.addFive(5));
        assertEquals(5, Example1.addFive(0));
        assertEquals(11, Example1.addFive(6));
    }
}

Example1.java

public class Example1 
{
    public static int addFive(int number)
    {
        number += 5;
        return number;
    }
}

Learners write a method that takes 5 int values as parameters and returns the average value of the 5 ints as a double. The method must be named average() and it must have 5 int parameters. The method must return a double.

Calling average(1, 5, 7, 4, 10) would return 5.4.

Assertions with doubles must have delta value (tolerance when comparing values).

import org.junit.Test;
import static org.junit.Assert.*;

public class CodingRoomsUnitTests {
    @Test
    public void testDefaultCase() {
        // You may rename this method to better suit the purpose of your test case
        // Your test case logic here
        assertEquals(5.4, Example2.average(1, 5, 7, 4, 10), 0.001);
        assertEquals(9.6, Example2.average(5, 9, 24, 6, 4), 0.001);
    }
}

Example2.java

public class Example2
{
    public static double average(int a, int b, int c, int d, int e)
    {
        //Calculate sum of 5 numbers
        int s = a + b + c + d + e;
        //cast s to double and divide by 5
        return (double) s / 5;
    }
}

Learners write a method that takes a String parameter. If the String has a double letter (i.e. contains the same letter twice in a row) then it should return true. Otherwise, it should return false.

This method must be named hasRepeat(String str) and have a String parameter. This method must return a boolean.

import org.junit.Test;
import static org.junit.Assert.*;

public class CodingRoomsUnitTests {
    @Test
    public void testDefaultCase() {
        // You may rename this method to better suit the purpose of your test case
        // Your test case logic here
        assertTrue(Example3.hasRepeat("mississippi"));
        assertFalse(Example3.hasRepeat("capsized"));
        assertTrue(Example3.hasRepeat("mazzone"));
        assertFalse(Example3.hasRepeat("this"));
    }
}

Example3.java

public class Example3
{
    public static boolean hasRepeat(String str)
    {
        for(int i = 0; i < str.length()-1; i++)
        {
            if(str.substring(i, i+1).equals(str.substring(i+1, i+2)))
            {
                return true;
            }
        }
        return false;
    }
}

This page shows how you can do some auto-grading of classes that only have void methods (methods that do not return a value) and a .toString() method.

This is completed by testing the output stream produced by the methods as they are called.

In this example, the comparison is completed with all character converted to lowercase and all spaces removed. If you want a pure character for character match, remove the .toLowerCase().trim().replace(" ", "") from the assertion.

Learners had to create an Employee class that has the following instance variables:

• String employee’s name

• int their 4 digit id number

• double hourly salary

Then, create a constructor method that sets each instance variable to the parameter values given to the constructor call.

Also, they must create a toString() method that prints the Employee's name, 4 digit id, and salary in the following format:

John Doe -- 4545 -- $15.67 per hour

Lastly, they must create a method called getRaise(double amount) that increases the employee's salary by the specified amount and prints “Nice you got a $” + amount + “ raise!”

For example, if we print the Employee object, call getRaise(1.0), and print the Employee object again, the following would be created.

John Doe -- 4545 -- $15.67 per hour 
Nice you got a $1.0 raise! 
John Doe -- 4545 -- $16.67 per hour

To auto-grade this, use the following -- you can run as many tests as your want!

First, there are two tests that ensure the .toString() method is working properly. They create an object with the Employee class, print the object, and ensure the print stream matches the expected output (remember, this example compares the two strings as all lowercase and with spaces removed).

Second, the .getRaise() method is tested with two different tests. The object is instantiated, .getRaise() is called, and then the object is printed to confirm that it effected the salary field. Note that that the print stream is reset between .getRaise() and printing the object so that each print statement can be tested asserted separately but within the same test.

import org.junit.Before;
import org.junit.After;
import org.junit.Test;
import static org.junit.Assert.*;
import java.io.*;


public class CodingRoomsUnitTests {
    
    private final PrintStream standardOut = System.out;
    private final ByteArrayOutputStream outputStreamCaptor = new ByteArrayOutputStream();

    @Before
    public void setUp() {
        System.setOut(new PrintStream(outputStreamCaptor));
    }

    @After
    public void tearDown() {
        System.setOut(standardOut);
    }

    @Test
    public void testToString1() {
        Employee e = new Employee("John Doe", 4545, 15.67);
        System.out.println(e);
        String answer = "John Doe -- 4545 -- $15.67 per hour".toLowerCase().trim().replace(" ", "");
        String output = outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "");
        assertEquals(answer, output);
    }

    public void testToString2() {
        Employee e = new Employee("Bob Bobbington", 5555, 10.5);
        System.out.println(e);
        String answer = "Bob Bobbington -- 5555 -- $10.5 per hour".toLowerCase().trim().replace(" ", "");
        String output = outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "");
        assertEquals(answer, output);
    }

    public void testGetRaise1() {
        Employee e = new Employee("Bob Bobbington", 5555, 10.5);
        e.getRaise(1);
        assertEquals(outputStreamCaptor.toString().toLowerCase().trim().replace(" ", ""), "Nice you got a $1.0 raise!".toLowerCase().trim().replace(" ", ""));
        System.setOut(standardOut);
        System.setOut(new PrintStream(outputStreamCaptor));
        System.out.println(e);
        String answer = "Bob Bobbington -- 5555 -- $11.5 per hour".toLowerCase().trim().replace(" ", "");
        String output = outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "");
        assertEquals(answer, output);
    }

    public void testGetRaise2() {
        Employee e = new Employee("Bob Bobbington", 5555, 10.5);
        e.getRaise(0.4);
        assertEquals(outputStreamCaptor.toString().toLowerCase().trim().replace(" ", ""), "Nice you got a $0.4 raise!".toLowerCase().trim().replace(" ", "");
        System.setOut(standardOut);
        System.setOut(new PrintStream(outputStreamCaptor));
        System.out.println(e);
        String answer = "Bob Bobbington -- 5555 -- $10.9 per hour".toLowerCase().trim().replace(" ", "");
        String output = outputStreamCaptor.toString().toLowerCase().trim().replace(" ", "");
        assertEquals(answer, output);
    }   
}

public class Employee 
{ 
    private String name; 
    private int id; 
    private double salary;
    
    public Employee(String n, int i, double s) 
    {
        name = n;
        id = i;
        salary = s;
    }
    public String toString() 
    {
        return name + " -- " + id + " -- $" + salary + " per hour";
    } 

    public void getRaise(double amount)
    {
        salary += amount;
        System.out.println("Nice you got a $" + amount + " raise!");
    }
}

Often you just want to make sure learners successfully used a particular function/command/structure. The simplest way I have found to do this is just parse the learner's code file and make sure what you are looking for is there.

For example, maybe you want to make sure the learner used println() at least 2 times. We can parse the learner's code file and count how many times "System.out.println(" is present in their code.

This test assumes the learner's file name is Example.java.

import org.junit.Test;
import static org.junit.Assert.*;
import java.nio.file.*;

public class CodingRoomsUnitTests {

    @Test
    public void testDefaultCase() throws Exception {
        int count = 0;
        String learner_code = new String(Files.readAllBytes(Paths.get("Example.java")));
        String[] words = learner_code.split(" ");

        for (int i = 0; i < words.length; i++) {
            if (words[i].indexOf("System.out.println(") >= 0) {
                count++;
            }
        }
        assertTrue(count >= 2);
    }
}

HTML/CSS Unit Tests

Autograding of HTML or CSS only works in the “HTML/CSS/JS“ language option.

The library we will use to check the HTML or CSS code is a Python library named htmlcssgrade. So, we must change the Unit Testing Framework from the default “JavaScript Jest (NodeJS)“ to “Python 3 unittest“.

Use the following as a starter template for your tests.

import unittest
import htmlcssgrade

class CodingRoomsUnitTests(unittest.TestCase):

    def test_check_html(self):
        learner_html = htmlcssgrade.HTML_Check("index.html")
        self.assertTrue()

    def test_check_css(self):
        learner_css = htmlcssgrade.CSS_Check("style.css")
        self.assertTrue()

Be sure to modify the template so the HTML_Check() and CSS_Check point to the file you want to check. In the example above HTML_check( ) points to index.html and CSS_Check points to style.css.

Inside the self.assertTrue() statements you can call any htmlcssgrade checks you want. You can have a sequence of assertTrue() statements to check as many parts of the learner’s file(s) you want.

import unittest
import htmlcssgrade

class CodingRoomsUnitTests(unittest.TestCase):

    def test_check_html(self):
        learner_html = htmlcssgrade.HTML_Check("index.html")
        self.assertTrue(learner_html.check_elements_attribute("img", "src", "img/hello.png"))
        self.assertTrue(learner_html.check_num_element_used("h2", 4))
        self.assertTrue(learner_html.check_element_content_exact("title", "Super Awesome Title"))

    def test_check_css(self):
        learner_css = htmlcssgrade.CSS_Check("css/style.css")
        self.assertTrue(learner_css.check_declaration("body", "background-color: #F06543"))
        self.assertTrue(learner_css.check_num_selector_rulesets(12))
        self.assertTrue(learner_css.check_selector_has_ruleset("p"))

For more options, read the docs below.

Full documentation and source can be found here. 

To start using the library in your code, be sure to import it

import htmlcssgrade

The first class that can be instantiated is used to check an HTML file with HTML_Check(). The constructor has one argument, filepath which is the path to the file you want to check.

Example:

learner_html = htmlcssgrade.HTML_Check("index.html")

Properties:

• filepath - the path to the file you are checking.

• html_obj - the BeautifulSoup object being used.

• code - text representation of the HTML file, which can be printed or parsed.

Methods:

check_HTML(code_snip) - Given a snip of HTML code, returns True if the snip is found in the learner's file.

Example:

learner_html.check_HTML('<meta charset="UTF-8"/>') 
learner_html.check_HTML("<p>This is a paragraph</p>")

check_element_used(element) - Given an element, returns True if the learner used the element.

Example:

learner_html.check_element_used("p") 
learner_html.check_element_used("ol")

check_num_element_used(element, number) - Given an element and number, returns True if learner's file has at least specified number of that element.

Example:

learner_html.check_num_element_used("h2", 4) 
learner_html.check_num_element_used("p", 6)

get_num_element_used(element) - Given an element, returns the number of times the element is used in learner's file.

Example:

learner_html.get_num_element_used("h2") 
learner_html.get_num_element_used("p")

check_element_content(element, content) - Given an element and content, returns True if the content is in the element (ignores capitalization, whitespace, etc).

Example:

learner_html.check_element_content("a", "Home") 
learner_html.check_element_content("title", "Super Awesome Title")

check_element_content_exact(element, content) - Given an element and content, returns True if the content is in the element character for character.

Example:

learner_html.check_element_content_exact("a", "Home") 
learner_html.check_element_content_exact("title", "Super Awesome Title")

check_elements_attribute(element, attribute, value) - Given an element, attribute, and value returns True if the element's attribute is equal to the value.

Example:

learner_html.check_elements_attribute("a", "href", "#") 
learner_html.check_elements_attribute("img", "src", "img/hello.png")

check_element_has_attribute(element, attribute) - Given an element and attribute, returns True if the element's attribute was assigned any value.

Example:

learner_html.check_element_has_attribute("img", "src") 
learner_html.check_element_has_attribute("a", "href")

get_list_of_elements_with_class(class_name) - Given a class name, returns a list of all elements with class set to given name.

Example:

learner_html.get_list_of_elements_with_class("center") 
learner_html.get_list_of_elements_with_class("gold")

check_element_has_class(element, class_name) - Given an element and class name, returns True if element was assigned class with given name.

Example:

learner_html.check_element_has_class("img", "center") 
learner_html.check_element_has_class("p", "center")

get_element_with_id(id_name) - Given an id name, returns the element assigned the id.

Example:

learner_html.get_element_with_id("container") 
learner_html.get_element_with_id("container")

check_element_has_id(element, id_name) - Given an element and id name, returns True if that element was assigned the id.

Example:

learner_html.check_element_has_id("div", "container") 
learner_html.check_element_has_id("p", "container")

check_use_css_file(css_filepath) - Given a CSS filepath, returns True if HTML code uses that CSS file.

Example:

learner_html.check_use_css_file("css/style.css") 
learner_html.check_use_css_file("blah.css")

check_use_js_file(js_filepath) - Given a JS filepath, returns True if HTML code uses that JS file.

Example:

learner_html.check_use_js_file("index.js") 
learner_html.check_use_js_file("blah.js")

extract_style_as_CSS_obj() - No parameters. Returns css_obj of all CSS found in <style> tags in the HTML file/text.

Example:

learner_css = learner_html.extract_style_as_CSS_obj()

learner_css = htmlcssgrade.CSS_Check("css/style.css")

learner_css.check_declaration(".center", "display: block") 
learner_css.check_declaration("body", "background-color: #F06543")

learner_css.check_property_used("body", "background-color") 
learner_css.check_property_used("body", "font-style")

learner_css.check_selector_rule("body", "background-color", "#F06543") 
learner_css.check_selector_rule("p", "font-style", "16pt")

learner_css.check_selector_has_ruleset("body") 
learner_css.check_selector_has_ruleset("p")

learner_css.get_selector_ruleset("body") 
learner_css.get_selector_ruleset(".center")

learner_css.get_property_value("body", "background-color") 
learner_css.get_property_value("p", "font-style")

learner_css.check_num_selector_declarations("body", 3) 
learner_css.check_num_selector_declarations("nav ul", 1)

learner_css.check_num_selector_declarations_equal("body", 3) 
learner_css.check_num_selector_declarations_equal("nav ul", 1)

learner_css.get_num_selector_declarations("body") 
learner_css.get_num_selector_declarations("nav ul")

learner_css.check_num_selector_rulesets(12) 
learner_css.check_num_selector_rulesets(20)

learner_css.check_num_selector_rulesets_equal(12) 
learner_css.check_num_selector_rulesets_equal(20)

learner_css.get_num_selector_rulesets()

learner_css.check_num_declarations(37)

learner_css.check_num_declarations_equal(37)

learner_css.get_num_declarations()

JavaScript Unit Tests

In Advanced zyLabs, you can use Jest, Jasmine, and Qunit as frameworks for autograding.

Jest - https://jestjs.io/

Jasmine - https://jasmine.github.io/

Qunit - https://api.qunitjs.com/

Qunit + Pupeteer - https://github.com/ameshkov/node-qunit-puppeteer

Below, are instructions for getting started with these test types in the JavaScript (Node.js) and HTML/CSS/JS image.

Imagine you have learners write a function named myFunction() which accepts 2 arguments and returns argument 1 times argument 2. It might look something like this.

function myFunction(p1, p2) {
  return p1 * p2;  
}
console.log(myFunction(2, 2));

You can easily test this function with Jest. For example, we can use the .toEqual matcher to see if the learner’s function returns the expected value.

const submission = require('./index.js');

test('testing learner myFunction() which takes 2 arguments and returns the product', () => {
    expect(submission.myFunction(2, 2)).toEqual(4);
    expect(submission.myFunction(5, 2)).toEqual(10);
    expect(submission.myFunction(100, 1)).toEqual(100);
});

Jest has a number of matchers, which can be found here: https://jestjs.io/docs/using-matchers

Now, in order for your test to use the learner’s function, you need to export it. So your learner’s JS file must have module.exports at the end.

For example:

function myFunction(p1, p2) {
  return p1 * p2;  
}
console.log(myFunction(2, 2));

module.exports = { myFunction };

You can do the same with Jest on HTML/CSS/JS. Just with one small difference when exporting modules.

Here is an example website you might want to test where a learner develops a function to convert Fahrenheit to Celsius.

index.html

<!DOCTYPE html>
<html>
    <body>
        <script type="text/javascript" src="index.js"></script>
        <h2>JavaScript Functions</h2>
        <p>This example calls a function to convert from Fahrenheit to Celsius:</p>
        <br>
        <form onsubmit="toCelsius(tempBox.value); return false">
            <label for="tempBox">Temp in F:</label> 
            <input type="text" name="tempBox" id="tempBox" />
            <input type="submit" />
        </form>
        <br>
        <p id="result"></p>
    </body>
</html>

index.js

function toCelsius(f) {
  temp = (5/9) * (f-32);
  document.getElementById("result").innerHTML = f + "°F is equal to " + temp + "°C";
  return temp
}

The Jest test is easy, we set up our matchers and test that the learner function works as expected.

const toCelsius = require('./index.js');

test('testing learner toCelsius() which takes a temp in F and returns C', () => {
    expect(toCelsius(100)).toEqual(37.77777777777778);
    expect(toCelsius(20)).toEqual(-6.666666666666667);
    expect(toCelsius(0)).toEqual(-17.77777777777778);
});

Now, in order for your test to use the learner’s function, you need to export it. So your learner’s JS file must have module.exports at the end. This might cause errors on your webpage as it is only needed for the server-side tests, so use a try-catch statement like the following.

function toCelsius(f) {
  temp = (5/9) * (f-32);
  document.getElementById("result").innerHTML = f + "°F is equal to " + temp + "°C";
  return temp
}

try {
  module.exports = toCelsius;
} catch (err) {
  console.log('toCelsius not loaded: ' + err);
}

Of course, you can add as many test cases as you want to test the learner’s code!

Here is an example checking that toCelsius is a function:

const toCelsius = require('./index.js');

test('testing learner toCelsius() to ensure it is a function.', () => {
    expect(typeof toCelsius).toEqual('function');  
});

Qunit with JavaScript (Node.js)

Qunit uses assertions to check whether a certain condition is true, such as assert.equal, assert.ok, assert.strictEqual, etc.

Every Qunit test includes a comment at the top to set the configuration. The default is // TYPE = module which uses the ES6 module syntax and is widely supported for modern web development. It provides clean syntax that can be written both server-side and client-side.

In the example below, we're using the configuration // TYPE = commonjs which provides a modular system for organizing code into separate files and uses require to import functionality from one file into another. It's also dynamically loaded and allows circular dependencies.

// TYPE = commonjs

const printSum = require('./index.js');

QUnit.test('Test with two numbers', function(assert) {

   let actualOutput = '';

   console.log = function(output) { actualOutput += `${output}`; };

   const num1 = [4, 6, 10, 2.5];
   const num2 = [2, 5.7, 4, 19];

   for (let i = 0; i < num1.length; i++) {
      const x = num1[i];
      const y = num2[i];
      const expectedOutput = `Sum is ${x + y}.`;
      printSum(x, y);
      assert.equal(actualOutput, expectedOutput, `Test printSum("${x}", "${y}")`);
      actualOutput = '';
   }
});

This test verifies the printSum function produces the correct output when given pairs of numbers as input. The QUnit assertions check if the actual output matches the expected output.

The AMD module can be used for asynchronous loading of modules, UMD for compatibility with CommonJS/AMD/global exports, etc.

Qunit + Puppeteer with HTML/CSS/JS

QUnit with Puppeteer provides automated tests in a headless browser environment. Web applications can be tested for simulated interaction in a realistic browser context.

// Set unit test configuration by changing the value in comment:
// 
// SOURCE_FILE = index.html
// BROWSER_HEADLESS = true

QUnit.test('Test form tag', function(assert) {
   let form = document.querySelector('form');
   assert.ok(form !== null, '<form> element exists');
   assert.equal(form.method.toUpperCase(), 'POST', 'Form method is POST');
   assert.equal(form.enctype, 'multipart/form-data', 'Form enctype is multipart/form-data');
   assert.equal(form.action, 'https://wp.zybooks.com/form-viewer.php', 'Form action is zybooks.com URL');
});

This test ensures the HTML form in theSOURCE_FILE =index.htmlaligns with the form attributes asserted in the test.

Jupyter Notebook (Python) Tests

Advanced zyLabs allow Jupyter Notebook autograding by converting the notebook to a Python script. We do this using Jupyter’s nbconvert tool.

When the IDE runs a test on Jupyter Notebook, it automatically tries to convert your default file to a Python file and replaces any dashes (-) in the file name with underscores (_).

Example: hello-world.ipynb would convert to hello_world.py

Advanced zyLabs only convert the default file. You would need to convert any additional notebook files you need to test. See below for more details.

Because the Notebook is converted to a Python script, an input/output comparison can be used with Jupyter Notebook.

For example, learners write something like this in Jupyter Notebook:

Their code can be tested with an input/output comparison like this:

For more information about input/output comparison tests, see the Test Bench section here.

Unit tests can be used in a similar way since Coding Rooms autograder converts your notebook to a Python file. Learn more about Python unit tests here.

Imagine you have learners write a function named add_five() that takes an integer as a parameter and returns 5 plus the parameter value. It might look like this in their notebook.

We can write a unit test case to check that the learner’s solution is correct.

import unittest

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import return_function
        self.assertEqual(10, return_function.add_five(5))
        self.assertEqual(5, return_function.add_five(0))
        self.assertEqual(11, return_function.add_five(6))

In the example above we assume the notebook’s name is return-function.ipynb. Which means Coding Rooms converted the file to return_function.py and line 6 in the above code is import return_function because that is the learner’s Python file name.

Convert notebooks with unit test script with just a few lines of Python code.

First, import the os module:

import os

Then, convert the notebook to a python

os.system("jupyter nbconvert --to script return-function.ipynb 2>/usercode/convertError")

The conversion has an output that goes into stderr and will fail the test if not directed to a file like 2>/usercode/convertError. If the test is not working properly, you may want to remove 2>/usercode/convertError to see if the conversion is not properly working.

Lastly, rename the file if it contains dashes, as Python does not allow dashes in module import names.

os.system("mv return-function.py return_function.py")

Do not rename the file to the same name as mv will throw an error. Omit this step if the file does not need to be renamed.

For example, if the above unit test needed its notebook converted, it would look like this:

import unittest
import os

os.system("jupyter nbconvert --to script return-function.ipynb 2>/usercode/convertError")
os.system("mv return-function.py return_function.py")

class CodingRoomsUnitTests(unittest.TestCase):

    def test_default_case(self):
        import return_function
        self.assertEqual(10, return_function.add_five(5))
        self.assertEqual(5, return_function.add_five(0))
        self.assertEqual(11, return_function.add_five(6))

Custom bash script tests can also be used to test the notebook.

Including being able to write your own jupyter nbconvert --to script commands, run Python unit tests, and and more.