Software testing

Testing is crucial in software development. If software is not tested, it is bound to fail, or at least have bugs after changes. Even carefully tested software has some faults. I tested Anki Books really carefully and still discover subtle bugs every once in a while.

RSpec tests of the FixOrdinalPositionsJob

I have had several times introduced slightly incorrect logic that resulted in articles having notes that had an invalid ordinal position set. For example, an article with notes at positions 0, 1, and 3 instead of 0, 1, and 2. Because I cannot figure out an efficient way to prevent this from ever happening, I developed a class called FixOrdinalPositionsJob:

##
# Job to fix ordinal positions in case they become incorrect due to a bug
class FixOrdinalPositionsJob < ApplicationJob
  ##
  # Fixes ordinal positions of all the articles' basic notes and books' articles
  def perform
    Book.find_each do |b|
      unless b.correct_children_ordinal_positions?
        b.articles.order(:ordinal_position).each_with_index do |articl, i|
          articl.update!(ordinal_position: i)
        end
      end
    end

    Article.find_each do |a|
      unless a.correct_children_ordinal_positions?
        a.notes.order(:ordinal_position).each_with_index do |note, i|
          note.update!(ordinal_position: i)
        end
      end
    end
  end
end

It has a comment above it that is pretty consistent with my paragraph above. I never know what to call these comments that are written above functions and extracted from the code as literate documentation. In the context of Ruby, you could call them RDoc comments and in general I think I've heard docstrings, function signatures, and I think they're good for public API documentation. Avoid using them for private API documentation. Anyway, you may notice that books also have ordinally positioned children (articles). This class can be invoked with the perform_now class method because it is a derived application job. It is a class that inherits that interface from the parent class, and by implementing its perform instance method, the perform_now method will call the perform method for you. This is related to the Active Job part of Rails. By the way, the logic of the above job is only correct because of a check constraint enforced on the PostgreSQL database that ordinal position integers cannot be negative.

Anki Books uses RSpec which is a good Ruby library (multiple gems) for testing that emphasizes the tests as specifications (RSpec tests are called specs a lot). Here are some RSpec tests for the above job class:

# frozen_string_literal: true

RSpec.describe FixOrdinalPositionsJob, ".perform" do
  subject(:fix_ordinal_positions_job) { described_class.perform_now }

  context "when a book has one article with an incorrect ordinal positions" do
    let(:book) { create(:book) }

    before do
      create_list(:article, 2, book:)
      bad_ord_pos_article = book.articles.second
      bad_ord_pos_article.ordinal_position = 2
      bad_ord_pos_article.save(validate: false)
    end

    it "fixes the ordinal positions" do
      expect(book.correct_children_ordinal_positions?).to be false
      fix_ordinal_positions_job
      expect(book.correct_children_ordinal_positions?).to be true
    end
  end

  # ... a lot of specs are omitted
end

Cucumber tests of the drag and drop functionality

Anki Books also has an automated feature test suite that uses Cucumber, Capybara, Selenium, :rack_test, chromedriver, etc. that allows writing the tests in plain English to drive the web browser through the app.

# Anki Books, a note-taking app to organize knowledge,
# is licensed under the GNU Affero General Public License, version 3
# Copyright (C) 2023 Kyle Rego

@javascript
Feature: Reordering basic notes

  Background:
    Given there is a user "test_user", email "test@example.com", and password "1234asdf!!!!"
    And the user "test_user" has a book called "test book 1"
    And the book "test book 1" has the article "test article 1"
    And the article "test article 1" has 5 basic notes
    And I am logged in as the user "test_user" with password: "1234asdf!!!!"
    And I am viewing the article "test article 1"

  Scenario: Reordering one basic note with drag and drop
    When I drag the note at position "0" to the dropzone at position "3"
    Then the front of the note at position "3" should be "Front of note 0"
    When I refresh the page
    Then the front of the note at position "3" should be "Front of note 0"

  Scenario: Reordering notes several times to reverse the original order
    When I drag the note at position "4" to the dropzone at position "0"
    And I drag the note at position "1" to the dropzone at position "4"
    And I drag the note at position "3" to the dropzone at position "1"
    And I drag the note at position "3" to the dropzone at position "2"
    Then the front of the note at position "0" should be "Front of note 4"    
    And the front of the note at position "1" should be "Front of note 3"
    And the front of the note at position "2" should be "Front of note 2"
    And the front of the note at position "3" should be "Front of note 1"
    And the front of the note at position "4" should be "Front of note 0"
    When I refresh the page
    Then the front of the note at position "0" should be "Front of note 4"
    And the front of the note at position "1" should be "Front of note 3"
    And the front of the note at position "2" should be "Front of note 2"
    And the front of the note at position "3" should be "Front of note 1"
    And the front of the note at position "4" should be "Front of note 0"

Selenium tests of the top nav keyboard accessibility

To do even more thorough testing, Anki Books also has a .NET Selenium NUnit project that also uses an automated web browser to test the app.

namespace Tests;

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

using Tests.Extensions;
using Tests.Interfaces;

[TestFixture]
public class TopNav : AppTests
{
    [Test]
    public void Test()
    {
        driver.Navigate().GoToUrl("http://localhost:3000/");
        driver.Manage().Window.Size = new System.Drawing.Size(948, 1003);
        driver.TryToLoginWithClick("test@example.com", "1234asdf!!!!");
        driver.DanceOnTopNav();
    }
}

Here is the extension method DanceOnTopNav() of the IWebDriver interface, implemented by the ChromeDriver class that represents a driver that can be controlled using the Selenium WebDriver language bindings, used in the above:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Interactions;

namespace Tests.Extensions;

public static partial class ChromeDriverExtensions
{
    /// <summary>
    /// Tabs and shift tabs in a pattern over the top nav
    /// </summary>
    /// <param name="driver"></param>
    public static void DanceOnTopNav(this IWebDriver driver)
    {
        driver.PressTabUntilOnText("Logout");
        driver.PressShiftTabUntilOnText("Read");
        driver.PressTabUntilOnText("Downloads");
        driver.PressShiftTabUntilOnText("Read");
        driver.PressTabUntilOnText("Books");
        driver.PressShiftTabUntilOnText("Read");
        driver.PressTabUntilOnText("Concepts");
        driver.PressShiftTabUntilOnText("Read");
        driver.PressTabUntilOnText("Write");
        driver.PressShiftTabUntilOnText("Read");
    }
}

At every level, tests prevent bugs before they occur (it is better for mental health to spend time testing instead of debugging), prevent bugs before they damage data and annoy users, drive the design (test-driven design) because you need to use interface of the code in the test code, and also provide documentation that is somewhat more reliably correct by it being passing tests.

There can be downsides with testing too if it is not done in a way that the benefits are worth more than the maintenance costs of the tests. Manual testing is error-prone but also time consuming. Other ways that testing can bring downsides are when the test suite takes forever to run which slows development, the short-term time investment of writing tests, and time spent maintaining tests that break naturally as the app refactors.

The best approach to testing is to automate as much as possible. This is mainly done by writing test code, code that tests the code.

In general, tests should ensure that the correct output or behavior happens for valid inputs, and that the software behaves appropriately for invalid inputs. User inputs should not be trusted to be valid. Sometimes even the application database cannot be trusted to always have valid data. Unit tests are good to make sure the app is having acceptable behavior for exceptional conditions. The heavy-duty tests that drive a web browser just need to focus on the happy path for the most part.

Summary: Test!

Article notes

What word is a common synonym for a fault?

Good programming leads to more time spent doing what than what?

Why is testing early a good idea?

According to Code Craft, studies show that there are how many errors per 1000 SLOC in carefully tested software?

What advocates that test code is written before the code being tested?

According to Code Craft, what is the golden rule of testing?

According to Code Craft, how often should tests be run?

What will happen in macro level testing if micro level testing is not thorough?

What is the comment to put at the top of a Ruby source code file to enforce frozen string literals?