Paper tutorial / Research in practice

Predicting students’ AI adoption with PLSpredict and CVPAT

Compare held-out Intention and Use predictions with the mean and linear benchmarks.

16 min read

We use this paper to show how a similar analysis can be carried out in QuickPLS. Follow the steps with the supplied practice data, then adapt the setup and calculations to your own study. The practice data are a separate teaching example.

A model can explain a sample and still predict poorly

Goto and Ramnarain studied culture, acceptance and AI use among 102 South African students. Their PLS analysis included a separate prediction assessment. The reported prediction comparisons were more favourable against a mean-only benchmark than against a linear-model benchmark.

In QuickPLS you will fit the adoption model, run PLSpredict / CVPAT and compare predictions for held-out students with mean-only and linear benchmarks. The original teaching data practise the paper’s distinction between fitted explanation and prediction; their errors and conclusions are separate from the paper’s sample and model.

Your study: what could you predict for a new student?

Imagine a university asking whether peer belonging and practical support help predict students’ AI-use intentions and reported use. The practice file contains 250 invented students. Each of four reflective constructs has three complete continuous teaching indicators from 1 to 7.

Create Belonging, Support, Intention and Use. Draw Belonging → Intention, Support → Intention, Intention → Use and Support → Use. For the first prediction task, imagine that Belonging and Support have already been measured for a new student, while Intention and Use have not.

That information rule matters. You cannot give the model a student’s actual intention answers while claiming to predict use from earlier belonging and support alone. The later exercise will deliberately change that information rule and name it as a different prediction task.

The model behind the prediction. Use these indicator blocks and arrows before running the additional analysis.
Use these indicator blocks and arrows before running the additional analysis.

On a small screen, swipe the table sideways to see every column.

ConstructColumnsAvailable in the first prediction task?
Belongingbelong1, belong2, belong3Yes
Supportsupport1, support2, support3Yes
Intentionintend1, intend2, intend3No; prediction target
Useuse1, use2, use3No; prediction target
Identifiercase_idCase label only

Step 1. Import, build and assess the original model

  1. Import the file

    In a writable project choose File → Import Data… → Choose data file…. In Review & import use raw data, Header row 1, comma separator and Decimal convention Dot (1.25). Keep measurement columns Continuous, choose Import dataset and verify 250 cases.

  2. Create the model and blocks

    Choose File → New Model… for the imported Dataset version. Ctrl-select each matching set of three indicators and choose Create construct. Name the four blocks in Properties → Model and choose Reflective under Parameter → Measurement model.

  3. Draw the four relationships

    Use Connect to draw Belonging and Support to Intention, then Intention and Support to Use. Return to Select, keep case_id unassigned and save.

  4. Fit and check the model

    Choose Calculate… → PLS-SEM Algorithm with Starting weights → Standard (+1) and Missing data → Treatment → Casewise deletion. Review Readiness and start. Inspect Outer loadings, Construct reliability and validity, HTMT+ and Inner VIF values before using the model for prediction.

Step 2. Keep fitted relationships separate from prediction errors

Read the original Direct effects and R-squared outputs to understand the fitted model. A positive Support → Intention path describes an adjusted relationship in these cases. Intention R-square describes fitted variation accounted for by its predictors. Neither value measures the error the model makes on a new student.

Cross-validation divides cases into folds. Each fold takes a turn as the held-out test group while the model is fitted to the other cases. The held-out target answers are used to assess prediction, not to train the model. Means, scaling and other preparation must also be learned from the training cases alone.

The teaching file is complete. If you later work with missing values, do not fill them using the whole dataset’s means before validation. That would let the held-out cases influence preparation. Choose a suitable training-only procedure within the prediction workflow and document it.

Step 3. Run a prediction plan with a clear information rule

Before starting, check Earliest antecedents, the stated folds, repetitions and seed against this lesson’s plan. After completion, select the matching result and read Prediction validation plan before the error tables. The panel calculates both IA and LM comparisons; there is no separate benchmark-selection dropdown.

  1. Open the prediction method

    With the saved model open choose Calculate…. Type PLSpredict / CVPAT in Find a method and select it. Open Methods first if the list is collapsed.

  2. Choose the available information

    Set Predictor information to Earliest antecedents. This matches the task in which Belonging and Support are known while Intention and Use are not. The model must predict intermediate information rather than use held-out intention answers as known.

  3. Set the teaching plan

    Choose Cross-validation folds 5, Repetitions 2, CVPAT confidence level 95% and Seed 202611080. Record Weighting scheme and Result data. Set Missing data to Casewise deletion for this complete file. These are practice choices, not a claim that five folds are always best.

  4. Calculate and preserve the plan

    Review Readiness, keep Open Results when finished selected and choose Start calculation. Open Model and data → Prediction validation plan, and save its folds, repetitions, seed and case information with the result.

    What you should see: The saved validation plan identifies the chosen file, held-out cases and earliest-antecedents task.

Screenshot to be addedSet the student prediction task

Check prediction on cases left out of training

Use five folds, two repetitions, seed 202611080 and the earliest-antecedents rule. Belonging and Support are known when predicting Intention and Use. For the Use outcome, the model must use predicted Intention rather than the held-out person’s observed Intention.

Every indicator below has 500 held-out predictions from 250 people, each predicted once in each repetition. Lower root mean squared error (RMSE) is better. The mean benchmark uses the training mean; the linear benchmark uses the permitted predictor indicators.

In this file, all six PLS prediction errors are below both benchmarks and every Q²predict value is positive. That is a result for this fixed prediction task and these teaching data; it does not establish usefulness in a new population.

About these answers: Predictions from QuickPLS calculation code were checked against independent cSEM training fits and prediction calculations in R.

On a small screen, swipe the table sideways to see every column.

Target indicatorPLS RMSEMean RMSELinear RMSEQ²predict
intend11.0311.0931.0480.109
intend21.0371.1011.0550.113
intend31.0351.1041.0540.122
use11.0611.0931.0830.057
use21.0601.1021.0870.075
use31.0741.0951.1010.039
Training boundaries and comparison scope

The saved plan identifies every training and test case. Means, standard deviations, PLS weights, structural coefficients and indicator reconstruction were estimated using training cases only. The same held-out cases and information rule were used for both benchmarks.

The check compared the QuickPLS 5x2 repeated prediction and six-indicator CVPAT calculation with independent cSEM training fits and prediction-loss calculations in R. The saved fold assignments were verified for both information rules. This used calculation code, not the installed desktop app.

Repeated predictions for the same person do not create 500 independent people. The paired loss test averages repetitions and target indicators within each of the 250 people.

Step 4. Compare one target with both benchmarks

Open Model fit → Indicator prediction summary. Select a named indicator such as use1 and compare its PLS error with IA and LM on the same scale. IA is the indicator-average benchmark: it predicts the training group’s average. LM is a linear-model benchmark using the information allowed by the task.

RMSE, root mean squared error, gives larger mistakes more influence. MAE, mean absolute error, is the average absolute size of a mistake. Lower error is better for the named target and measure. Read every target rather than selecting only the favourable indicator.

If one benchmark is unavailable, inspect its status and warnings. For example, redundant predictors can prevent a unique linear-model fit within a training fold. Missing LM output is not evidence that PLS beats LM. Keep a valid IA comparison under its own name.

Hypothetical example

Illustrative reading example

Suppose use1 has RMSE 0.90 for PLS, 1.10 for IA and 0.84 for LM. PLS improves on the mean-only prediction but makes larger errors than LM. The predictors carry useful information, but this row does not show an advantage from the PLS model over the linear comparison. These values are hypothetical.

Screenshot to be addedCompare errors for the six target indicators

Step 5. Read the error difference in the right direction

Open Inference → CVPAT benchmark assessment (single model). CVPAT is the cross-validated predictive ability test. It compares paired prediction losses for the same cases. The calculation reports PLS loss minus benchmark loss, so a negative difference favours PLS.

Read IA and LM separately, with the named loss, target set, uncertainty, probability and availability status. The setup does not provide a separate loss picker; use the Loss label in the result. An aggregate comparison concerns the selected targets together. Indicator-level comparisons ask narrower questions and are not automatically adjusted for examining many indicators.

Two repetitions do not turn 250 students into 500 independent participants. The procedure combines repeated predictions for each case before inference. More repetitions can reduce variation from the split plan, but they cannot add independent information or fix a sampling design that does not represent future users.

Hypothetical example

Illustrative reading example

In a separate arithmetic example, PLS squared errors total 81 and IA squared errors total 100. Q²_predict is 1 - 81/100 = 0.19. This positive value concerns improvement over IA only; it says nothing by itself about LM. If the IA error total were zero, the ratio would be undefined. These are illustrative totals, not the preceding RMSE row or fitted answers.

Screenshot to be addedRead the paired prediction comparison

Read the paired loss difference

The difference is PLS squared error minus benchmark squared error. A negative value favours PLS. The p-value is one-sided for lower PLS loss; the displayed interval is a two-sided 95% t interval based on 250 case-level differences. Individual-indicator comparisons in the download are separate, unadjusted tests.

About these answers: Aggregate CVPAT reference for the earliest-information prediction task.

On a small screen, swipe the table sideways to see every column.

BenchmarkMean loss difference95% intervalOne-sided p-value
IA-0.104-0.174 to -0.0330.002112
LM-0.046-0.072 to -0.0200.000242

Step 6. Try a later prediction time without calling it a free improvement

Save a separate analysis using Predictor information → Direct antecedents. Keep the same folds, repetitions and seed. For Use, the held-out student’s Intention indicators are now available as predictors, alongside Support. This corresponds to a later prediction point at which intention has already been measured.

Compare the target-specific errors, but state that the information changed. A lower Use error in this analysis would not prove that the original early prediction improved. It would show the value of having additional measurements available under the second task.

In a real project, ask whether those measurements would exist when a decision must be made. If collecting intention requires another survey after the intervention decision, using it in the earlier task would be unrealistic. A clear information rule makes the prediction claim useful to the reader.

Compare the later information rule honestly

The direct-antecedents run may use the held-out person’s observed Intention when predicting Use. It answers a later prediction question. The Intention prediction task is unchanged, but an improvement for Use would come with additional information. Do not present it as a fairer model winning the original task.

On a small screen, swipe the table sideways to see every column.

Target indicatorPLS RMSEMean RMSELinear RMSEQ²predict
intend11.0311.0931.0480.109
intend21.0371.1011.0550.113
intend31.0351.1041.0540.122
use11.0291.0931.0580.113
use21.0321.1021.0640.123
use31.0551.0951.0840.072

A disappointing prediction can be the correct finding

On a small screen, swipe the table sideways to see every column.

ObservationCheckAction
High fitted R-square but large held-out errorTraining fit versus test predictionsReport weak prediction; a good training fit does not overturn it.
PLS beats IA but not LMBoth benchmark rows on the same targetsState both findings instead of calling prediction strong in general.
LM is unavailableRedundant predictors, fold size and warningsInvestigate availability and do not claim superiority over the missing comparison.
Some indicators improve and others worsenThe full target table and aggregate comparisonReport the pattern rather than selecting favourable indicators.
A negative loss difference is uncertainIts interval, probability and independent case countDescribe uncertain evidence of an advantage; do not search seeds.
A later-information run improvesEarliest versus direct antecedent ruleLabel it as a different prediction task.

Revise the model without training on the validation result

If prediction is weak, identify a substantive reason for revision: poorly measured inputs, missing information or unnecessary complexity. Keep the original result and label any search through alternatives as exploratory. Repeatedly trying models against the same validation results gradually uses those results as training information.

For a final predictive claim, use a fresh suitable assessment after model development. If cases are time ordered or clustered within classes, ordinary shuffled folds may not represent the intended future prediction problem. This independent synthetic exercise does not resolve those design choices for a real student study.

State what was known before stating how well you predicted

Methods outline: We predicted Intention and Use indicators from earliest-antecedent Belonging and Support information using [fold count] folds repeated [count] times with seed [value]. Preparation was learned from training cases. We compared PLS errors with IA and LM and read paired CVPAT results at [confidence level]. We retained [independent case count] and recorded [availability and failures].

Results outline: For [target], PLS RMSE was [value], compared with [IA] and [LM]. The aggregate PLS-minus-benchmark loss difference was [value and uncertainty] against IA and [value and uncertainty] against LM. [Explain disagreements across targets.] The Direct antecedents analysis used additional intention information and is reported as a separate task.

ReportInclude
Prediction questionTargets and information available at the intended decision time.
Validation planFolds, repetitions, seed, case selection and row order.
PreparationTraining-only transformations and missing-data handling.
ErrorsNamed indicator, loss measure and both available benchmarks.
CVPATSubtraction direction, uncertainty, target level and independent count.
RevisionsExploratory changes and any fresh assessment used afterward.

Write a prediction claim that states what was known

Using five folds repeated twice with earliest antecedents, the model produced lower held-out RMSE than both the training-mean and linear benchmarks for all six target indicators in the 250-case teaching file. Aggregate paired squared-error differences favoured PLS against both benchmarks.

Report the seed and fold plan, the information available at prediction time, all target errors and the unit used in the paired test. Keep the later direct-antecedents analysis separate.

Explain the predictions for this practice file

  • Compare PLS, training-mean and linear-model RMSE for all six target indicators. State which method has the smaller error in each row.
  • Explain why the aggregate PLS-minus-benchmark loss differences favour PLS in this file, using their intervals and the stated test direction.
  • Distinguish the 250 independent students from the 500 held-out predictions per indicator.
  • State why held-out Intention is unavailable in the earliest-antecedents task and available in the separate direct-antecedents task.
  • Report the two prediction tasks separately. Keep conclusions limited to these synthetic data and the stated information rules.
  • Optional contrast: explain how a model could improve on the mean benchmark while performing worse than the linear benchmark, using the separate hypothetical example.

Further reading

Related guides