Quickcheck
1 Quickstart
1.1 Installation
1.2 Running tests
2 Running checks
quickcheck
quickcheck-results
check
check-results
3 Configuration, results, and utilities
config
with-small-test-count
with-medium-test-count
with-large-test-count
with-test-count
result
property
property?
testable?
4 Integration with Rack  Unit
check-property
check-property/  config
add-property-check-info
5 Generators
generator
choose-integer
choose-real
choose-ascii-char
choose-ascii-letter
choose-printable-ascii-char
choose-char
choose-list
choose-vector
choose-string
choose-symbol
choose-one-of
choose-mixed
choose-with-frequencies
generator-unit
generator-bind
generator-sequence
sized
arbitrary?
arbitrary
arbitrary-boolean
arbitrary-char
arbitrary-ascii-char
arbitrary-printable-ascii-char
arbitrary-integer
arbitrary-natural
arbitrary-rational
arbitrary-real
arbitrary-mixed
arbitrary-one-of
arbitrary-pair
arbitrary-list
arbitrary-vector
arbitrary-tuple
arbitrary-record
arbitrary-string
arbitrary-ascii-string
arbitrary-printable-ascii-string
arbitrary-symbol
arbitrary-procedure
6 Operations on properties
==>
label
classify
trivial
collect
7 Random number generation
make-random-generator
random-generator?
random-generator-next
random-generator-split
random-integer
random-real
6.7

Quickcheck

Mike Sperber
and Ismael Figueroa

This manual provides documentation for a Racket implementation of Quickcheck, a library that tests the specification of a program with randomly generated test inputs.

The original Quickcheck is a library for Haskell.

 (require quickcheck) package: quickcheck

1 Quickstart

1.1 Installation

If you have not installed the package yet, run the following command from the command-line:

  raco pkg install quickcheck

Note that this package will require Racket v6.0 or newer to install.

Alternatively, you may use the GUI package manager from DrRacket to install the package.

1.2 Running tests

To use Quickcheck, you first write a specification for a program as a property in ordinary Racket code. For example, the following is a property that expresses the English specification “the string->number function produces numbers when given strings”:

> (define string->number-returns-number
    (property ([str arbitrary-string])
      (number? (string->number str))))

Given such a property, we can run random tests using quickcheck:

> (quickcheck string->number-returns-number)

Falsifiable, after 0 tests:

str = "\u0002\u0003\u0002"

You may have already guessed that this property will be easily falsified, since most strings do not actually parse as numbers.

Next, let’s write a property that we expect will actually succeed. For example, the following encodes the English specification “given two lists of integers, the result of appending them will have the same length as the sum of the original list lengths”:

> (define append-and-length-agree
    (property ([lst-1 (arbitrary-list arbitrary-integer)]
               [lst-2 (arbitrary-list arbitrary-integer)])
      (= (+ (length lst-1) (length lst-2))
         (length (append lst-1 lst-2)))))

Testing the property reveals that it holds up:

> (quickcheck append-and-length-agree)

OK, passed 100 tests.

2 Running checks

procedure

(quickcheck prop)  void?

  prop : testable?
Test the given prop, generating new test cases if prop specifies a generator.

Prints the result of the check, including a counterexample for the property if one was found.

Example:
> (quickcheck (property ((str arbitrary-string))
                (string=? str (list->string (string->list str)))))

OK, passed 100 tests.

procedure

(quickcheck-results prop)

  
integer? stamps (or/c boolean? result?)
  prop : testable?
Like quickcheck, except it will return three values instead:

Examples:
> (define-values (ntests stamps ok?)
    (quickcheck-results (property ((str arbitrary-string))
                          (string=? str (list->string (string->list str))))))
> ntests

100

> (first stamps)

'()

> ok?

#t

procedure

(check config prop)  void?

  config : config?
  prop : testable?
Runs a check like quickcheck using the given config as a configuration.

Prints the result of the check, including a counterexample for the property if one was found.

procedure

(check-results config prop)  result?

  config : config?
  prop : testable?
Like check, except it will return three result values instead of printing the results. See the quickcheck function for an explanation of the return values.

3 Configuration, results, and utilities

struct

(struct config (max-test max-fail size print-every)
    #:extra-constructor-name make-config)
  max-test : number?
  max-fail : number?
  size : (-> integer? integer?)
  print-every : (-> integer? (listof any/c) any)
A structure type that represents configurations for the test process. An instance of this type must be supplied when calling the check or check-results functions.

The max-test field represents the maximum number of succeeding tests that will be run. The max-fail field represents the maximum number of tests that can be run with no result before the checker terminates.

The size field should be a function of one argument that produces the test size given the current test number. The print-every field should be a function that takes the test number and the generated arguments and is called for its side effect.

The quickcheck and quickcheck-results functions use a default config where the test count is 100, the test size for test n is (+ 3 (quotient n 2)), the max fail count is ten times the test count, and nothing is printed. The default config can be adjusted to run different numbers of tests with with-small-test-count, with-medium-test-count, and with-large-test-count.

syntax

(with-small-test-count body ...)

Within body ..., the number of test cases used by the quickcheck functions is 100.

Example:
> (with-small-test-count
    (quickcheck (property ((str arbitrary-string))
                  (string=? str (list->string (string->list str))))))

OK, passed 100 tests.

syntax

(with-medium-test-count body ...)

Within body ..., the number of test cases used by the quickcheck functions is 1000.

Example:
> (with-medium-test-count
    (quickcheck (property ((str arbitrary-string))
                  (string=? str (list->string (string->list str))))))

OK, passed 1000 tests.

syntax

(with-large-test-count body ...)

Within body ..., the number of test cases used by the quickcheck functions is 10000.

Example:
> (with-large-test-count
    (quickcheck (property ((str arbitrary-string))
                  (string=? str (list->string (string->list str))))))

OK, passed 10000 tests.

syntax

(with-test-count test-count-expr body ...)

Within body ..., the number of test cases used by teh quickcheck functions is test-count-expr.

Example:
> (with-test-count 42
    (quickcheck (property ((str arbitrary-string))
                  (string=? str (list->string (string->list str))))))

OK, passed 42 tests.

struct

(struct result (ok stamp arguments-list)
    #:extra-constructor-name make-result)
  ok : (or/c null #t #f)
  stamp : (listof string?)
  arguments-list : (listof any/c)
Represents a single result from a test. The ok field is #t on success, #f on failure, and null if there is no test result.

The stamp field represents the labels that were relevant to this test execution. The arguments-list is a list of the values generated for checking this test case.

syntax

(property ([id gen/arb-expr] ...) body0 body ...)

 
  gen/arb-expr : (or/c arbitrary? generator?)
Constructs a testable property for functions like quickcheck.

The ids are bound to the result of the given gen/arb-exprs inside the body expressions. The body expressions are used as the bodies of a predicate function that will be run with newly generated values from the specified generators or arbitraries.

Example:
> (property ((str arbitrary-string))
    (string=? str (list->string (string->list str))))

#<property>

procedure

(property? val)  boolean?

  val : any/c
Returns #t if val is a property for testing, otherwise returns #f.

procedure

(testable? val)  boolean?

  val : any/c
Returns #t if val is a value that can be tested with functions like quickcheck. Returns #f otherwise.

Values that can be tested are the following:

4 Integration with RackUnit

By default quickcheck simply displays whether the property check results in success or, in case of failure, it prints the arguments that falsify the property along with some other metadata. This is not helpful for (semi-) automatic testing as it is done using RackUnit. Fortunately it is possible to combine the best of both worlds.

 (require rackunit/quickcheck) package: quickcheck

syntax

(check-property prop)

Like quickcheck but a RackUnit exception is raised if the property fails. The exception is raised using fail-check and it is augmented with additional data: the number of tests performed, the stamps, and the arguments that falsify the property.

For example, let us check the previous string->number-returns-number property:

> (check-property string->number-returns-number)

--------------------

FAILURE

ntest:      0

stamps:     ()

arguments:  "str = \"\\u0002\\u0003\\u0002\""

name:       check-property

location:   (eval 16 0 16 1)

expression: (check-property string->number-returns-number)

params:     (#<property>)

Falsifiable

--------------------

syntax

(check-property/config config prop)

Similar to check-property but taking a specific config object.

syntax

(add-property-check-info ((name value) ...))

Adds specific check-info data when checking a property, using either check-property or check-property/config. Its usage is like with-check-info but it does not define a new scope. Instead, it modifies an internal parameter that is cleared at every usage of check-property or check-property/config.

The purpose of this form is to improve error messages when a property fails. For instance, to compare whether two functions are observationally equivalent for arbitrary arguments, we can define the following form using add-property-check-info to state the expected and actual values inside the declaration of a property:

(define-syntax-rule (conforms-to f g ([name gen] ...))
    (property ([name gen] ...)
              (let ([expected (f name ...)]
                    [actual (g name ...)])
                (add-property-check-info (['expected expected]
                                          ['actual actual]))
                (equal? expected actual))))
 
(define (f n) (+ n 1))
(define (g n) (- n 1))
> (check-property (conforms-to f g ([n arbitrary-integer])))

--------------------

FAILURE

expected:   1

actual:     -1

ntest:      0

stamps:     ()

arguments:  "n = 0"

name:       check-property

location:   (eval 20 0 20 1)

expression: (check-property (conforms-to f g ((n arbitrary-integer))))

Falsifiable

--------------------

Crucially, every time the property is tested with a different set of arguments add-property-check-info overrides the previous data.

5 Generators

struct

(struct generator (proc)
    #:extra-constructor-name make-generator)
  proc : (-> integer? random-generator? any/c)
Represents a source of values for randomly testing a given property with functions like quickcheck.

The proc value should be a function that accepts an integer representing the size of the value, a random number generator, and returns a value for testing.

procedure

(choose-integer lower upper)  generator?

  lower : integer?
  upper : integer?
Produces a generator that returns integers between lower and upper.

procedure

(choose-real lower upper)  generator?

  lower : real?
  upper : real?
Produces a generator that returns real numbers between lower and upper.

A generator that returns ASCII characters.

A generator that returns ASCII letter characters.

A generator that returns ASCII characters that have a printed representation.

procedure

(choose-char lower upper)  generator?

  lower : char?
  upper : char?
Produces a generator that returns characters with integer values between lower and higher.

procedure

(choose-list elem-gen size)  generator?

  elem-gen : generator?
  size : integer?
Produces a generator that returns lists of the given size and with elements generated with elem-gen.

procedure

(choose-vector elem-gen size)  generator?

  elem-gen : generator?
  size : integer?
Produces a generator that returns vectors of the given size and with elements generated with elem-gen.

procedure

(choose-string char-gen size)  generator?

  char-gen : generator?
  size : integer?
Produces a generator that returns strings of the given size and with elements generated with elem-gen, which must generate characters.

procedure

(choose-symbol char-gen size)  generator?

  char-gen : generator?
  size : integer?
Produces a generator that returns symbols of the given size and with elements generated with elem-gen, which must generate characters.

procedure

(choose-one-of opts)  generator?

  opts : (listof any/c)
Produces a generator that returns one of the values in the list opts.

procedure

(choose-mixed promises)  generator?

  promises : (listof (promise/c generator?))
Produces a generator that returns returns values from one of the generator promises, forcing the promise when it does so.

procedure

(choose-with-frequencies freqs)  generator?

  freqs : (listof (cons/c integer? generator?))
Produces a generator that returns a value from one of the given generators, weighing each generator by the matching integer in freqs.

procedure

(generator-unit val)  generator?

  val : any/c
Produces a generator that always returns the given val.

procedure

(generator-bind gen k)  generator?

  gen : generator?
  k : (-> any/c generator?)
Produce a generator from the result of calling k on the value generated from the generator gen.

procedure

(generator-sequence gens)  generator?

  gens : (listof generator?)
Given the list of generators gens, produce a generator that produces values from them in sequence.

procedure

(sized f)  generator?

  f : (-> integer? generator?)
Produces a generator from a function f that constructs a generator given an integer representing a value’s size.

Recognizes arbitrarys.

procedure

(arbitrary gen trans)  arbitrary?

  gen : generator?
  trans : (-> any/c generator? generator?)
Returns a source of randomly generated values, except where the values are filtered by the function trans to produce a narrower set of new values.

Generates a boolean value.

Generates a character.

Generates an ASCII character.

Generates a printable ASCII character.

Generates an integer.

Generates a non-negative integer.

Generates a rational number.

Generates a real number.

procedure

(arbitrary-mixed pred+promises)  arbitrary?

  pred+promises : (listof (cons/c (-> any/c any/c) (promise/c arbitrary?)))
Produces a arbitrary generator given a list matching up predicates to promises that produce arbitrary generators.

procedure

(arbitrary-one-of eql? vals)  arbitrary?

  eql? : (any/c any/c -> any/c)
  vals : (listof any/c)
Produces an arbitrary generator that generates values from the list of values vals. The values are filtered by the equality function eql?.

procedure

(arbitrary-pair fst rst)  arbitrary?

  fst : arbitrary?
  rst : arbitrary?
Produces an arbitrary generator that generates pairs of values drawn from the fst and rst generators respectively.

procedure

(arbitrary-list elem)  arbitrary?

  elem : arbitrary?
Produces an arbitrary generator that generates lists in which the element values are drawn from elem.

procedure

(arbitrary-vector elem)  arbitrary?

  elem : arbitrary?
Produces an arbitrary generator that generates vectors in which the element values are drawn from elem.

procedure

(arbitrary-tuple elem ...)  arbitrary?

  elem : arbitrary?
Produces an arbitrary generator that generates constant-length lists in which the element values are drawn from the generators elems in order.

procedure

(arbitrary-record constructor    
  accessors    
  elem ...)  arbitrary?
  constructor : procedure?
  accessors : (listof procedure?)
  elem : arbitrary?
Produces an arbitrary generator that generates values of some structure type given a constructor procedure and accessor procedures for that structure type. The values passed to the constructor are drawn from the generators elems.

Generates a string.

Generates an ASCII string.

Generates an ASCII string consisting of characters with printable representations.

Generates a symbol.

procedure

(arbitrary-procedure result arg ...)  arbitrary?

  result : arbitrary?
  arg : arbitrary?
Generates a procedure that takes arguments drawn from the generators args and which produces a value drawn from the generator result.

6 Operations on properties

syntax

(==> bool-expr prop)

Represents implication for testable properties.

If bool-expr is #t, equivalent to prop. Otherwise, produces a property that returns no result.

procedure

(label str test)  generator?

  str : string?
  test : testable?
Labels the given test with a str label to help identify what portion of the property is failing.

syntax

(classify really? label-expr testable-expr)

Labels the result of testable-expr with the value of label-expr if really? evaluates to #t. Otherwise just returns the result of testable-expr.

syntax

(trivial really? testable-expr)

Like classify, but always uses the label "trivial".

procedure

(collect lbl test)  generator?

  lbl : any/c
  test : testable?
Labels the given test with the written form of the value lbl. Similar to the label function.

7 Random number generation

procedure

(make-random-generator s1 s2)  random-generator?

  s1 : number?
  s2 : number?
Constructs a random number generator, given two seeds s1 and s2.

procedure

(random-generator? val)  boolean?

  val : any/c
Returns #t if val is a random number generator constructed by make-random-generator and returns #f in all other cases.

procedure

(random-generator-next rand)  
integer? random-generator?
  rand : random-generator?
Produce a random integer and a random number generator.

procedure

(random-generator-split rand)

  
random-generator? random-generator?
  rand : random-generator?
Splits the given random number generator and returns two random number generators.

procedure

(random-integer rg low high)  integer?

  rg : random-generator?
  low : integer?
  high : integer?
Returns a random integer between the bounds low and high.

procedure

(random-real rg low high)  real?

  rg : random-generator?
  low : real?
  high : real?
Returns a random real number between the bounds low and high.