Test suites

View docs

Test suites lie at the heart of psycholinguistic evaluation. The items in a test suite are given as input to a language model, and the resulting surprisal values are used to assess the model's performance. Typically, test suites are designed in a way that probes a particular grammatical phenomenon.

Browse the available test suites in the table below, or add a new test suite by creating one interactively or uploading one as a .json file.

Available test suites
Name Reference Added by Models evaluated Average performance
Name Reference Added by Models evaluated Average performance
position_object_gap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO
position_pp_gap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO
position_pp_nogap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO
position_subject_gap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO
position_object_nogap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO
position_subject_nogap James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James TODO TODO