Interests

• Web development
• Property-based testing
• Formal methods

Idea: Mash it up!

• Combine linear temporal logic (LTL) with a functional language
• Leverage the DOM and introspective capabilities
• Run as property-based tests

Goals

• Tester should focus on specifying and understanding
• No more sleep or wait
• Support partial specifications

Non-goals

• Deterministic testing
  • Optional shrinking
• Support for specific frameworks

How It Works

1. Navigate to origin page
2. Record a trace (sequence of states and actions):
   1. Generate random actions
   2. Pick one possible action and perform it
   3. Record state
   4. Go to 2.1 if not done
3. Check that the behavior (only the states) satisfies the specification
4. If rejected, shrink sequence of actions and rerun

Early benchmark: TodoMVC

• Use TodoMVC as a test for Quickstrom
• Wrote a single spec (for new and legacy formats)
• Checked mainstream implementations
• Found issues in Angular and Mithril implementations
• Submitted a GitHub issue with findings

The TodoMVC Showdown

• Improved the specification
• Checked all implementations
• Results:
  • 37 passed
  • 12 failed
  • 4 not testable
• Wrote a blog post detailing the results

Specification Language

• Based on PureScript
• Extended with:
  • Linear temporal logic operators
  • DOM queries
• Use regular PureScript packages
• Interpreter built in Haskell

Temporal Operators

• Change the modality of the sub-expression

• Available operators from LTL:

  next :: forall a. a -> a
  always :: Boolean -> Boolean
  until :: Boolean -> Boolean -> Boolean

DOM Queries

• Two operators:
  • queryOne
  • queryAll
• Take as arguments:
  1. CSS selector
  2. Record of element state specifiers

Example: DOM Queries

queryAll "button" { textContent, disabled }
    :: Array { textContent :: String
             , disabled :: Bool 
             }

queryOne "input" { value }
    :: Maybe { value :: String }

Actions

• We must instruct Quickstrom which actions to try

• List of weighted probabilities and action specifiers

  Array (Tuple Int Action)

• Comes with predefined actions, e.g. foci, clicks

• Often need to carefully pick selectors, actions, and weights

Specification Structure

module Spec where

import Quickstrom

readyWhen = ".my-app"

actions = clicks -- or something else

proposition = initial 
  && always (transition1 || transition2 || ...)

State Transitions

transition1 =
  (something == "foo")
  && next (something == "bar")

PureScript Support

• Some supported PureScript packages:
  • numbers
  • strings
  • arrays
  • transformers
  • generics-rep
• FFI is implemented in Haskell
  • Packages’ foreign functions are built into Quickstrom

Running Tests

Use the check command:

quickstrom check Example.spec.purs http://example.com

Origin can be a file:

quickstrom check Example.spec.purs example.html

Test Failures

1. State
  • .play-pause
      -
         - property "textContent" = "Play"
  • .time-display
      -
         - property "textContent" = "00:00"
2. click button[0]
3. click button[0]
4. State
  • .play-pause
      -
         - property "textContent" = "Play"
  • .time-display
      -
         - property "textContent" = "NaN:NaN"

Failed after 1 tests and 4 levels of shrinking.

Cross-Browser Testing

• Currently supports two browsers:
  • Firefox
  • Chrome/Chromium
• Theoretically all WebDriver-enabled browsers

Trailing State Changes

• Default: record a single state after each action
• Override: also record trailing state changes
• Often caused by asynchronous operations
• Two command-line options:
  • --max-trailing-state-changes
  • --trailing-state-change-timeout

The Specification Language

• PureScript was a very good first choice
• A custom language is probably the next step
• Currently working together with Liam O’Connor

Possible Improvements

• Better error reporting
• Coverage (for specifications)
• Targeted search
• Screenshotting unique states

Commercial Product

• Keep Quickstrom and the CLI open source
• Build a SaaS on top
  • Browser-based IDE for specs and runner
  • Scheduled checks and reports
  • Integrations (CI, WebDriver services)

Resources

• quickstrom.io (main website)
• buttondown.email/quickstrom (newsletter)