Information Matrix Test

Daphna Harel

2017-04-05

This vignette reproduces the example shows the use of the information matrix test when collapsing categories of a Partial Credit Model.

library(IMTest)
library(lme4)
library(reshape2)

A commonly analyzed dataset in the illustration of IRT methods consists of 316 responses to 24 items referring to verbally aggressive reactions to situations that may be frustrating. Responses to the items are collected on a three point ordinal scale, with categories labelled as yes, perhaps, or no. The dataset is publicly available, for example through the lme4 package in R. This vignette analyzes item responses to the first 12 items, and investigates the potential model misspecification when collapsing categories of those items.

data(VerbAgg)

x = dcast(VerbAgg[,c( "id","item", "resp")], id~item)

## Using resp as value column: use value.var to override.

for(i in 2:25){
  x[,i] = factor(x[,i], levels = c("no", "perhaps", "yes"))
}

my_data = x[,2:13]

J = dim(my_data)[[2]]
n = dim(my_data)[[1]]

First, a set of index vectors is created to test both item-level parameters of each item separately.

# Create index vectors
ind = list()
for(i in 1:J){
  ind[[i]] = rep(0, J*2)
  ind[[i]][(2*i-1):(2*i)] = 1
}

A collapsing function is generated, showing that no collapsing has occurred, and the gpcm_IMT function is run. A matrix is initialized to store the p-values.

col = split(rep(c(1:3), J), rep(1:J, each = 3))
temp_data = collapse_data(my_data, col, "rasch")
mod = gpcm_IMT(temp_data$data, constraint = "rasch")

## Creates a matrix to store results.
pval = matrix(c(0), nrow = J, ncol = 3)

The following set of code runs the IMT for the uncollapsed data, and for two forms of collapsing: collapsing the middle category up, and collapsing the middle category down.

# No Collapsing
for(i in 1:J){
  test_fit = IMT(mod, "rasch", R = 5000, ind[[i]])
  pval[i,1] = pchisq(test_fit$Tstat, test_fit$df, lower.tail = F)
}

# Collapsing the middle categoy up.
for(i in 1:J){
  col = split(rep(c(1:3), J), rep(1:J, each = 3))
  col[[i]] = c(1,2,2)
  temp_data = collapse_data(my_data, col, "rasch")
  mod = gpcm_IMT(temp_data$data, constraint = "rasch")
  test_fit = IMT(mod, "rasch", R = 5000, temp_data$ind)
  pval[i,2] = pchisq(test_fit$Tstat, test_fit$df, lower.tail = F)
}

# Collapsing the middle category down.
for(i in 1:J){
  col = split(rep(c(1:3), J), rep(1:J, each = 3))
  col[[i]] = c(1,1,2)
  temp_data = collapse_data(my_data, col, "rasch")
  mod = gpcm_IMT(temp_data$data, constraint = "rasch")
  test_fit = IMT(mod, "rasch", R = 5000, temp_data$ind)
  pval[i,3] = pchisq(test_fit$Tstat, test_fit$df, lower.tail = F)
}

Originally, all but one item fails to reject goodness-of-fit. After collapsing in one direction, four items now reject goodness-of-fit. After collapsing in the other direction, no items reject goodness-of-fit.

# Original fit: all but question 8 fail to reject goodness-of-fit.
p.adjust(pval[,1], method = "fdr")

##  [1] 0.36900738 0.67580018 0.67580018 0.67631742 0.67580018 0.67631742
##  [7] 0.35523985 0.02169192 0.67580018 0.35523985 0.67580018 0.67580018

# After collapsing: question 8 now fails to reject goodness-of-fit, but 4 other questions reject.
p.adjust(pval[,2], method = "fdr")

##  [1] 0.01764817 0.01764817 0.31797817 0.02136294 0.01636977 0.34694767
##  [7] 0.22677140 0.16009954 0.63597791 0.66808694 0.08544049 0.79247415

# Had it been collapsed the other direction, all would fail to reject.
p.adjust(pval[,3], method = "fdr")

##  [1] 0.07919704 0.07871714 0.64556259 0.25786818 0.14567761 0.14567761
##  [7] 0.42430856 0.40091425 0.14567761 0.42430856 0.33813012 0.26181778