| Title: | Computerized Adaptive Testing of the Big Five with the Rating Scale Model |
|---|---|
| Description: | A 'shiny' application, built as a 'golem' package, that gives the 50 statements of the IPIP Big-Five Factor Markers as a computerized adaptive test (CAT). Each trait is scored with the rating scale model, the next statement is the one with the most Fisher information at the current expected a posteriori (EAP) estimate, and a trait stops when its standard error reaches a threshold. The app is the questionnaire as a respondent sees it, with a profile of the five traits at the end and a PDF report of it, made with 'typst'. The engine, which gives the same results as 'catR', and the session logic are also R functions, and any item bank fitted with the rating scale model of 'mirt' can be read. |
| Authors: | Dimitrios Zacharatos [aut, cre] |
| Maintainer: | Dimitrios Zacharatos <[email protected]> |
| License: | GPL-3 |
| Version: | 0.0.0.9000 |
| Built: | 2026-11-21 09:24:20 UTC |
| Source: | https://github.com/sedzinfo/golemcat |
Scores the answer (category answer - 1, or 5 - answer on a reversed
item), estimates the ability and its standard error again, and either selects the
next item of the scale or, when the scale stops, moves to the next scale or ends the
test.
cat_answer(s, answer)cat_answer(s, answer)
s |
A CAT session. |
answer |
The answer, 1 to 5, as shown to the person (before any reversal). |
The updated session.
Other CAT session:
cat_item(),
cat_results(),
cat_start()
s <- cat_start(ipip50()$bank, scales = "Openness") s <- cat_answer(s, 5) s$progresss <- cat_start(ipip50()$bank, scales = "Openness") s <- cat_answer(s, 5) s$progress
The item to show now
cat_item(s)cat_item(s)
s |
A CAT session. |
A list with the scale, the stem (NA when the bank has
none) and the question of the current item, or NULL when the test is
done.
Other CAT session:
cat_answer(),
cat_results(),
cat_start()
s <- cat_start(ipip50()$bank, scales = "Openness") cat_item(s)s <- cat_start(ipip50()$bank, scales = "Openness") cat_item(s)
The standardized ability of each scale, its standard error, and the percentile: the
share of the population of the calibration sample expected below that ability,
assuming a normal distribution (pnorm of the standardized ability).
cat_results(s)cat_results(s)
s |
A CAT session. |
A data frame, one row per scale: scale, theta and se
(standardized), percentile (0 to 100), answered, total and
status.
Other CAT session:
cat_answer(),
cat_item(),
cat_start()
s <- cat_start(ipip50()$bank, scales = c("Openness", "Neuroticism")) while (s$stage == "testing") s <- cat_answer(s, 2) cat_results(s)s <- cat_start(ipip50()$bank, scales = c("Openness", "Neuroticism")) while (s$stage == "testing") s <- cat_answer(s, 2) cat_results(s)
Starts a computerized adaptive test (CAT) over the chosen scales of an item bank,
one scale at a time, in the order given. Within a scale, each next item is the most
informative one left (next_item_rsm); after each answer the ability is
estimated again by EAP (eap_est_rsm), with the normal prior of the
scale, over a grid of prior standard deviations. A scale stops when the
standard error, divided by the prior standard deviation, is at most
se_threshold, or when its items run out; the test then moves to the next scale.
cat_start(bank, scales = unique(bank$scale), se_threshold = 0.45)cat_start(bank, scales = unique(bank$scale), se_threshold = 0.45)
bank |
An item bank read with |
scales |
The scales to test, in order. |
se_threshold |
Standardized standard error at which a scale stops. |
Abilities and standard errors are reported standardized per scale (divided by the
prior standard deviation), so the same threshold is the same target in every scale:
a threshold se aims at a reliability of about .
A CAT session: a list to pass to cat_answer,
cat_item and cat_results. Its stage is
"testing" until the last scale stops, then "done"; progress
has one row per scale and log one row per answer.
Other CAT session:
cat_answer(),
cat_item(),
cat_results()
q <- ipip50() s <- cat_start(q$bank, scales = "Extraversion", se_threshold = 0.45) while (s$stage == "testing") s <- cat_answer(s, 4) # always "Moderately accurate" cat_results(s) s$logq <- ipip50() s <- cat_start(q$bank, scales = "Extraversion", se_threshold = 0.45) while (s$stage == "testing") s <- cat_answer(s, 4) # always "Moderately accurate" cat_results(s) s$log
The density of the normal distribution, used as the prior of the EAP estimate.
It returns the same result as stats::dnorm().
density_function(x, mean = 0, sd = 1)density_function(x, mean = 0, sd = 1)
x |
Numeric vector of values at which the density is evaluated. |
mean |
Mean of the distribution. |
sd |
Standard deviation of the distribution. |
A numeric vector of densities.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
eap_est_rsm(),
eap_se_rsm(),
integrate_cat(),
likelihood_rsm(),
next_item_rsm()
v <- seq(-3, 3, by = 0.1) all.equal(density_function(v), dnorm(v))v <- seq(-3, 3, by = 0.1) all.equal(density_function(v), dnorm(v))
The expected a posteriori estimate of ability, the mean of the posterior, by
quadrature over nqp points from lower to upper. It returns the
same result as catR::eapEst(model = "RSM").
eap_est_rsm( bank, x, D = 1, priorPar = c(0, 1), lower = -4, upper = 4, nqp = 33 )eap_est_rsm( bank, x, D = 1, priorPar = c(0, 1), lower = -4, upper = 4, nqp = 33 )
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
x |
Vector of the answers, scored 0 to m, one per row of |
D |
Metric constant, 1 or 1.702. |
priorPar |
Mean and standard deviation of the normal prior. |
lower, upper
|
Range of the quadrature. |
nqp |
Number of quadrature points. |
The estimate, a number.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
density_function(),
eap_se_rsm(),
integrate_cat(),
likelihood_rsm(),
next_item_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) eap_est_rsm(bank, c(4, 3, 2, 1, 0))bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) eap_est_rsm(bank, c(4, 3, 2, 1, 0))
The posterior standard deviation when theta is the EAP estimate; at any other
value, the posterior root mean squared deviation around it. It returns the same
result as catR::eapSem(model = "RSM").
eap_se_rsm( theta, bank, x, D = 1, priorPar = c(0, 1), lower = -4, upper = 4, nqp = 33 )eap_se_rsm( theta, bank, x, D = 1, priorPar = c(0, 1), lower = -4, upper = 4, nqp = 33 )
theta |
The EAP estimate. |
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
x |
Vector of the answers, scored 0 to m, one per row of |
D |
Metric constant, 1 or 1.702. |
priorPar |
Mean and standard deviation of the normal prior. |
lower, upper
|
Range of the quadrature. |
nqp |
Number of quadrature points. |
The standard error, a number.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
density_function(),
eap_est_rsm(),
integrate_cat(),
likelihood_rsm(),
next_item_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) x <- c(4, 3, 2, 1, 0) eap_se_rsm(theta = eap_est_rsm(bank, x), bank = bank, x = x)bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) x <- c(4, 3, 2, 1, 0) eap_se_rsm(theta = eap_est_rsm(bank, x), bank = bank, x = x)
The Fisher information of each item at an ability value,
. Under the rating scale
model every item shares the same thresholds, so every information curve has the
same shape and differs only by a shift of . The test information is
the sum of the item information. It returns the same result as
catR::Ii(model = "RSM").
Ii_rsm(theta, bank, D = 1)Ii_rsm(theta, bank, D = 1)
theta |
Ability. |
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
D |
Metric constant, 1 or 1.702. |
A list with Ii, the information of each item.
Other rating scale model:
Pi_rsm(),
density_function(),
eap_est_rsm(),
eap_se_rsm(),
integrate_cat(),
likelihood_rsm(),
next_item_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) Ii_rsm(theta = 0, bank) sum(Ii_rsm(theta = 0, bank)$Ii) # test informationbank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) Ii_rsm(theta = 0, bank) sum(Ii_rsm(theta = 0, bank)$Ii) # test information
Integrates y = f(x) over the points x, as catR::integrate.catR() does.
integrate_cat(x, y)integrate_cat(x, y)
x |
Numeric vector of the points of integration, in increasing order. |
y |
Numeric vector of the values of the function at |
The integral, a number.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
density_function(),
eap_est_rsm(),
eap_se_rsm(),
likelihood_rsm(),
next_item_rsm()
x <- seq(from = -3, to = 3, length = 33) integrate_cat(x = x, y = exp(x)) exp(3) - exp(-3) # the exact valuex <- seq(from = -3, to = 3, length = 33) integrate_cat(x = x, y = exp(x)) exp(3) - exp(-3) # the exact value
The questionnaire built into the app: the 50 items of the IPIP Big-Five Factor
Markers (Goldberg, 1992; public domain), 10 per trait, with the coefficients of a
rating scale model fitted with mirt to each trait separately, on the
responses of 19,718 people to the Big Five test of the Open-Source Psychometrics
Project. The 18 reverse-keyed items were reversed (6 - answer) before the fit.
ipip50()ipip50()
A list: bank, the item bank (see read_item_bank), with
the traits in the order Openness, Conscientiousness, Extraversion, Agreeableness,
Neuroticism, and likert, the five answer options (see
read_likert).
Other item bank:
read_item_bank(),
read_likert(),
validate_item_bank_against_mirt()
q <- ipip50() table(q$bank$scale) q$likertq <- ipip50() table(q$bank$scale) q$likert
The product, over the items, of the probability of the category given, under local independence.
likelihood_rsm(theta, bank, x, D = 1)likelihood_rsm(theta, bank, x, D = 1)
theta |
Ability. |
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
x |
Vector of the answers, scored 0 to m, one per row of |
D |
Metric constant, 1 or 1.702. |
The likelihood, a number.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
density_function(),
eap_est_rsm(),
eap_se_rsm(),
integrate_cat(),
next_item_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) likelihood_rsm(theta = 0, bank = bank, x = c(4, 3, 2, 1, 0))bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) likelihood_rsm(theta = 0, bank = bank, x = c(4, 3, 2, 1, 0))
Picks, among the items not yet given, the one with the most information at the
current ability estimate. Under the rating scale model this is the item whose
is nearest to . With randomesque = 1 it returns
the same item as catR::nextItem(model = "RSM", criterion = "MFI").
next_item_rsm(theta, bank, out = NULL, D = 1, randomesque = 1)next_item_rsm(theta, bank, out = NULL, D = 1, randomesque = 1)
theta |
Current ability estimate. |
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
out |
Indices of the items already given. |
D |
Metric constant, 1 or 1.702. |
randomesque |
Number of most informative items to draw one from at random: 1 is pure maximum information, 3 to 5 spreads the exposure of the items. |
A list: item, the index of the selected item in the bank,
info, its information at theta, and available, the indices of
the items that were still available.
Other rating scale model:
Ii_rsm(),
Pi_rsm(),
density_function(),
eap_est_rsm(),
eap_se_rsm(),
integrate_cat(),
likelihood_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) next_item_rsm(theta = 0, bank) next_item_rsm(theta = 1.5, bank, out = c(4, 5)) # a short adaptive test responses <- c() administered <- c() theta <- 0 for (step in 1:4) { item <- next_item_rsm(theta, bank, out = administered)$item administered <- c(administered, item) responses <- c(responses, 4) # the answer, scored 0 to 4 theta <- eap_est_rsm(bank[administered, , drop = FALSE], responses) cat("step", step, "item", item, "theta", round(theta, 4), "se", round(eap_se_rsm(theta, bank[administered, , drop = FALSE], responses), 4), "\n") }bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) next_item_rsm(theta = 0, bank) next_item_rsm(theta = 1.5, bank, out = c(4, 5)) # a short adaptive test responses <- c() administered <- c() theta <- 0 for (step in 1:4) { item <- next_item_rsm(theta, bank, out = administered)$item administered <- c(administered, item) responses <- c(responses, 4) # the answer, scored 0 to 4 theta <- eap_est_rsm(bank[administered, , drop = FALSE], responses) cat("step", step, "item", item, "theta", round(theta, 4), "se", round(eap_se_rsm(theta, bank[administered, , drop = FALSE], responses), 4), "\n") }
The probability of each answer category, and its first derivative, at an ability value, under the rating scale model of Andrich (1978):
with the empty sum of category 0 equal to zero. It returns the same result as
catR::Pi(model = "RSM").
Pi_rsm(theta, bank, D = 1)Pi_rsm(theta, bank, D = 1)
theta |
Ability. |
bank |
Matrix of item parameters, one row per item: column 1 is the location
of the item |
D |
Metric constant, 1 or 1.702. |
A list: Pi, the probabilities, one row per item and one column per
category, and dPi, their first derivatives.
Other rating scale model:
Ii_rsm(),
density_function(),
eap_est_rsm(),
eap_se_rsm(),
integrate_cat(),
likelihood_rsm(),
next_item_rsm()
bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) Pi_rsm(theta = 0, bank)$Pi rowSums(Pi_rsm(theta = 0, bank)$Pi) # each item's probabilities add up to 1bank <- matrix(c(-0.560, -0.230, 1.559, 0.071, 0.129, rep(1.715, 5), rep(0.461, 5), rep(-1.265, 5), rep(-0.687, 5)), nrow = 5, dimnames = list(1:5, c("lambdaj", "delta1", "delta2", "delta3", "delta4"))) Pi_rsm(theta = 0, bank)$Pi rowSums(Pi_rsm(theta = 0, bank)$Pi) # each item's probabilities add up to 1
Reads the item bank of a questionnaire, fitted with the rating scale model of
mirt (itemtype = "rsm"), and converts the coefficients to the form of
the engine (Pi_rsm): and .
It stops with a message naming the problem when the file is not a complete rating
scale bank, so that an app does not start and then fail on the first answer.
read_item_bank(items_csv)read_item_bank(items_csv)
items_csv |
Path to the csv file, saved as UTF-8. |
The csv has one row per item, with the columns scale and question
(the text of the item); a1, b1 to b4 and c, the
coefficients of mirt; and prior_sd, the standard deviation of the
latent distribution of the scale, which mirt estimates. Optional columns:
stem, the instruction shown above the item, and reverse
(TRUE/FALSE), for items whose answers were reversed (6 - answer)
before the fit. Other columns are ignored.
A data frame, one row per item: scale, question, stem,
reverse, lambdaj, delta1 to delta4, prior_mean
(always 0) and prior_sd.
Other item bank:
ipip50(),
read_likert(),
validate_item_bank_against_mirt()
bank <- read_item_bank(system.file("extdata", "ipip50_items.csv", package = "golemcat")) head(bank[, c("scale", "question", "reverse", "lambdaj")])bank <- read_item_bank(system.file("extdata", "ipip50_items.csv", package = "golemcat")) head(bank[, c("scale", "question", "reverse", "lambdaj")])
Reads the answer options of a questionnaire: a csv with the columns value
(1 to 5) and label, from the lowest to the highest agreement. The answer
v is scored as category v - 1 of the rating scale model
(5 - v on reversed items), so with the 4 thresholds b1 to b4
there must be exactly the values 1 to 5.
read_likert(likert_csv)read_likert(likert_csv)
likert_csv |
Path to the csv file, saved as UTF-8. |
A data frame with the columns value and label, ordered by value.
Other item bank:
ipip50(),
read_item_bank(),
validate_item_bank_against_mirt()
read_likert(system.file("extdata", "ipip50_likert.csv", package = "golemcat"))read_likert(system.file("extdata", "ipip50_likert.csv", package = "golemcat"))
Writes the profile of a finished CAT session as a PDF: the five traits at a glance,
a page on each trait, and how the scores are calculated, with the numbers of the
session. It needs the 'typst' command line (see typst_command).
report_pdf(res, file, bank = ipip50()$bank, date = Sys.Date())report_pdf(res, file, bank = ipip50()$bank, date = Sys.Date())
res |
The results, from |
file |
Path of the PDF to write. |
bank |
The item bank, from |
date |
The date printed on the report. |
file, invisibly.
## Not run: q <- ipip50() s <- cat_start(q$bank, unique(q$bank$scale)) while (s$stage == "testing") s <- cat_answer(s, sample(1:5, 1)) report_pdf(cat_results(s), "profile.pdf", q$bank) ## End(Not run)## Not run: q <- ipip50() s <- cat_start(q$bank, unique(q$bank$scale)) while (s$stage == "testing") s <- cat_answer(s, sample(1:5, 1)) report_pdf(cat_results(s), "profile.pdf", q$bank) ## End(Not run)
Starts the app, an adaptive (CAT) test of the Big Five personality traits with the 50 statements of the IPIP Big-Five Factor Markers. It has two tabs:
The questionnaire as a respondent sees it: one statement at a time, and at the end a profile of the five traits, each as a percentile of the calibration sample, and a PDF report of the profile.
How the test works, the acknowledgments and the references.
The workings of an adaptive test, step by step and under six IRT models, are in the CAT tab of golemirt (https://sedzinfo-golemirt.share.connect.posit.cloud/).
run_app( se_threshold = 0.4, analytics = FALSE, keep_alive = TRUE, onStart = NULL, options = list(), enableBookmarking = NULL, uiPattern = "/", ... )run_app( se_threshold = 0.4, analytics = FALSE, keep_alive = TRUE, onStart = NULL, options = list(), enableBookmarking = NULL, uiPattern = "/", ... )
se_threshold |
Standardized standard error at which a trait stops in the Take the test tab. The default 0.40 aims at a reliability of about .84 per trait and uses about 43 of the 50 statements; 0.45 uses about 36. |
analytics |
|
keep_alive |
|
onStart |
A function that will be called before the app is actually run.
This is only needed for |
options |
Named options that should be passed to the |
enableBookmarking |
Can be one of |
uiPattern |
A regular expression that will be applied to each |
... |
Arguments passed on to |
A shiny.appobj: printed, it starts the app.
## Not run: run_app() run_app(se_threshold = 0.45, options = list(port = 8080)) ## End(Not run)## Not run: run_app() run_app(se_threshold = 0.45, options = list(port = 8080)) ## End(Not run)
Compares the response probabilities of the engine with those of the fitted
mirt model, item by item, so that a sign or mapping mistake does not go
unnoticed (a check on the first item alone passes either way, since mirt
fixes c = 0 there). Items are matched by name: make.names() of the
question must be the item name in the model. Needs the mirt package.
validate_item_bank_against_mirt( bank_scale, mirt_model, thetas = seq(-4, 4, 0.5), tol = 1e-06 )validate_item_bank_against_mirt( bank_scale, mirt_model, thetas = seq(-4, 4, 0.5), tol = 1e-06 )
bank_scale |
The rows of one scale of an item bank read with
|
mirt_model |
The |
thetas |
Ability values at which the probabilities are compared. |
tol |
Largest difference accepted. |
The largest difference, invisibly; a warning when it is above tol.
Other item bank:
ipip50(),
read_item_bank(),
read_likert()
## Not run: bank <- read_item_bank("items.csv") validate_item_bank_against_mirt(bank[bank$scale == "Openness", ], model_openness) ## End(Not run)## Not run: bank <- read_item_bank("items.csv") validate_item_bank_against_mirt(bank[bank$scale == "Openness", ], model_openness) ## End(Not run)