A model can explain a sample and still predict poorly
Goto and Ramnarain studied culture, acceptance and AI use among 102 South African students. Their PLS analysis included a separate prediction assessment. The reported prediction comparisons were more favourable against a mean-only benchmark than against a linear-model benchmark.
In QuickPLS you will fit the adoption model, run PLSpredict / CVPAT and compare predictions for held-out students with mean-only and linear benchmarks. The original teaching data practise the paper’s distinction between fitted explanation and prediction; their errors and conclusions are separate from the paper’s sample and model.
Your study: what could you predict for a new student?
Imagine a university asking whether peer belonging and practical support help predict students’ AI-use intentions and reported use. The practice file contains 250 invented students. Each of four reflective constructs has three complete continuous teaching indicators from 1 to 7.
Create Belonging, Support, Intention and Use. Draw Belonging → Intention, Support → Intention, Intention → Use and Support → Use. For the first prediction task, imagine that Belonging and Support have already been measured for a new student, while Intention and Use have not.
That information rule matters. You cannot give the model a student’s actual intention answers while claiming to predict use from earlier belonging and support alone. The later exercise will deliberately change that information rule and name it as a different prediction task.
On a small screen, swipe the table sideways to see every column.
| Construct | Columns | Available in the first prediction task? |
|---|---|---|
| Belonging | belong1, belong2, belong3 | Yes |
| Support | support1, support2, support3 | Yes |
| Intention | intend1, intend2, intend3 | No; prediction target |
| Use | use1, use2, use3 | No; prediction target |
| Identifier | case_id | Case label only |
Step 1. Import, build and assess the original model
Import the file
In a writable project choose File → Import Data… → Choose data file…. In Review & import use raw data, Header row 1, comma separator and Decimal convention Dot (1.25). Keep measurement columns Continuous, choose Import dataset and verify 250 cases.
Create the model and blocks
Choose File → New Model… for the imported Dataset version. Ctrl-select each matching set of three indicators and choose Create construct. Name the four blocks in Properties → Model and choose Reflective under Parameter → Measurement model.
Draw the four relationships
Use Connect to draw Belonging and Support to Intention, then Intention and Support to Use. Return to Select, keep case_id unassigned and save.
Fit and check the model
Choose Calculate… → PLS-SEM Algorithm with Starting weights → Standard (+1) and Missing data → Treatment → Casewise deletion. Review Readiness and start. Inspect Outer loadings, Construct reliability and validity, HTMT+ and Inner VIF values before using the model for prediction.
Step 2. Keep fitted relationships separate from prediction errors
Read the original Direct effects and R-squared outputs to understand the fitted model. A positive Support → Intention path describes an adjusted relationship in these cases. Intention R-square describes fitted variation accounted for by its predictors. Neither value measures the error the model makes on a new student.
Cross-validation divides cases into folds. Each fold takes a turn as the held-out test group while the model is fitted to the other cases. The held-out target answers are used to assess prediction, not to train the model. Means, scaling and other preparation must also be learned from the training cases alone.
The teaching file is complete. If you later work with missing values, do not fill them using the whole dataset’s means before validation. That would let the held-out cases influence preparation. Choose a suitable training-only procedure within the prediction workflow and document it.
Step 3. Run a prediction plan with a clear information rule
Before starting, check Earliest antecedents, the stated folds, repetitions and seed against this lesson’s plan. After completion, select the matching result and read Prediction validation plan before the error tables. The panel calculates both IA and LM comparisons; there is no separate benchmark-selection dropdown.
Open the prediction method
With the saved model open choose Calculate…. Type PLSpredict / CVPAT in Find a method and select it. Open Methods first if the list is collapsed.
Choose the available information
Set Predictor information to Earliest antecedents. This matches the task in which Belonging and Support are known while Intention and Use are not. The model must predict intermediate information rather than use held-out intention answers as known.
Set the teaching plan
Choose Cross-validation folds 5, Repetitions 2, CVPAT confidence level 95% and Seed 202611080. Record Weighting scheme and Result data. Set Missing data to Casewise deletion for this complete file. These are practice choices, not a claim that five folds are always best.
Calculate and preserve the plan
Review Readiness, keep Open Results when finished selected and choose Start calculation. Open Model and data → Prediction validation plan, and save its folds, repetitions, seed and case information with the result.
What you should see: The saved validation plan identifies the chosen file, held-out cases and earliest-antecedents task.
Check prediction on cases left out of training
Use five folds, two repetitions, seed 202611080 and the earliest-antecedents rule. Belonging and Support are known when predicting Intention and Use. For the Use outcome, the model must use predicted Intention rather than the held-out person’s observed Intention.
Every indicator below has 500 held-out predictions from 250 people, each predicted once in each repetition. Lower root mean squared error (RMSE) is better. The mean benchmark uses the training mean; the linear benchmark uses the permitted predictor indicators.
In this file, all six PLS prediction errors are below both benchmarks and every Q²predict value is positive. That is a result for this fixed prediction task and these teaching data; it does not establish usefulness in a new population.
About these answers: Predictions from QuickPLS calculation code were checked against independent cSEM training fits and prediction calculations in R.
On a small screen, swipe the table sideways to see every column.
| Target indicator | PLS RMSE | Mean RMSE | Linear RMSE | Q²predict |
|---|---|---|---|---|
| intend1 | 1.031 | 1.093 | 1.048 | 0.109 |
| intend2 | 1.037 | 1.101 | 1.055 | 0.113 |
| intend3 | 1.035 | 1.104 | 1.054 | 0.122 |
| use1 | 1.061 | 1.093 | 1.083 | 0.057 |
| use2 | 1.060 | 1.102 | 1.087 | 0.075 |
| use3 | 1.074 | 1.095 | 1.101 | 0.039 |
Training boundaries and comparison scope
The saved plan identifies every training and test case. Means, standard deviations, PLS weights, structural coefficients and indicator reconstruction were estimated using training cases only. The same held-out cases and information rule were used for both benchmarks.
The check compared the QuickPLS 5x2 repeated prediction and six-indicator CVPAT calculation with independent cSEM training fits and prediction-loss calculations in R. The saved fold assignments were verified for both information rules. This used calculation code, not the installed desktop app.
Repeated predictions for the same person do not create 500 independent people. The paired loss test averages repetitions and target indicators within each of the 250 people.
Step 4. Compare one target with both benchmarks
Open Model fit → Indicator prediction summary. Select a named indicator such as use1 and compare its PLS error with IA and LM on the same scale. IA is the indicator-average benchmark: it predicts the training group’s average. LM is a linear-model benchmark using the information allowed by the task.
RMSE, root mean squared error, gives larger mistakes more influence. MAE, mean absolute error, is the average absolute size of a mistake. Lower error is better for the named target and measure. Read every target rather than selecting only the favourable indicator.
If one benchmark is unavailable, inspect its status and warnings. For example, redundant predictors can prevent a unique linear-model fit within a training fold. Missing LM output is not evidence that PLS beats LM. Keep a valid IA comparison under its own name.
Hypothetical example
Illustrative reading example
Suppose use1 has RMSE 0.90 for PLS, 1.10 for IA and 0.84 for LM. PLS improves on the mean-only prediction but makes larger errors than LM. The predictors carry useful information, but this row does not show an advantage from the PLS model over the linear comparison. These values are hypothetical.
Step 5. Read the error difference in the right direction
Open Inference → CVPAT benchmark assessment (single model). CVPAT is the cross-validated predictive ability test. It compares paired prediction losses for the same cases. The calculation reports PLS loss minus benchmark loss, so a negative difference favours PLS.
Read IA and LM separately, with the named loss, target set, uncertainty, probability and availability status. The setup does not provide a separate loss picker; use the Loss label in the result. An aggregate comparison concerns the selected targets together. Indicator-level comparisons ask narrower questions and are not automatically adjusted for examining many indicators.
Two repetitions do not turn 250 students into 500 independent participants. The procedure combines repeated predictions for each case before inference. More repetitions can reduce variation from the split plan, but they cannot add independent information or fix a sampling design that does not represent future users.
Hypothetical example
Illustrative reading example
In a separate arithmetic example, PLS squared errors total 81 and IA squared errors total 100. Q²_predict is 1 - 81/100 = 0.19. This positive value concerns improvement over IA only; it says nothing by itself about LM. If the IA error total were zero, the ratio would be undefined. These are illustrative totals, not the preceding RMSE row or fitted answers.
Read the paired loss difference
The difference is PLS squared error minus benchmark squared error. A negative value favours PLS. The p-value is one-sided for lower PLS loss; the displayed interval is a two-sided 95% t interval based on 250 case-level differences. Individual-indicator comparisons in the download are separate, unadjusted tests.
About these answers: Aggregate CVPAT reference for the earliest-information prediction task.
On a small screen, swipe the table sideways to see every column.
| Benchmark | Mean loss difference | 95% interval | One-sided p-value |
|---|---|---|---|
| IA | -0.104 | -0.174 to -0.033 | 0.002112 |
| LM | -0.046 | -0.072 to -0.020 | 0.000242 |
Step 6. Try a later prediction time without calling it a free improvement
Save a separate analysis using Predictor information → Direct antecedents. Keep the same folds, repetitions and seed. For Use, the held-out student’s Intention indicators are now available as predictors, alongside Support. This corresponds to a later prediction point at which intention has already been measured.
Compare the target-specific errors, but state that the information changed. A lower Use error in this analysis would not prove that the original early prediction improved. It would show the value of having additional measurements available under the second task.
In a real project, ask whether those measurements would exist when a decision must be made. If collecting intention requires another survey after the intervention decision, using it in the earlier task would be unrealistic. A clear information rule makes the prediction claim useful to the reader.
Compare the later information rule honestly
The direct-antecedents run may use the held-out person’s observed Intention when predicting Use. It answers a later prediction question. The Intention prediction task is unchanged, but an improvement for Use would come with additional information. Do not present it as a fairer model winning the original task.
On a small screen, swipe the table sideways to see every column.
| Target indicator | PLS RMSE | Mean RMSE | Linear RMSE | Q²predict |
|---|---|---|---|---|
| intend1 | 1.031 | 1.093 | 1.048 | 0.109 |
| intend2 | 1.037 | 1.101 | 1.055 | 0.113 |
| intend3 | 1.035 | 1.104 | 1.054 | 0.122 |
| use1 | 1.029 | 1.093 | 1.058 | 0.113 |
| use2 | 1.032 | 1.102 | 1.064 | 0.123 |
| use3 | 1.055 | 1.095 | 1.084 | 0.072 |
A disappointing prediction can be the correct finding
On a small screen, swipe the table sideways to see every column.
| Observation | Check | Action |
|---|---|---|
| High fitted R-square but large held-out error | Training fit versus test predictions | Report weak prediction; a good training fit does not overturn it. |
| PLS beats IA but not LM | Both benchmark rows on the same targets | State both findings instead of calling prediction strong in general. |
| LM is unavailable | Redundant predictors, fold size and warnings | Investigate availability and do not claim superiority over the missing comparison. |
| Some indicators improve and others worsen | The full target table and aggregate comparison | Report the pattern rather than selecting favourable indicators. |
| A negative loss difference is uncertain | Its interval, probability and independent case count | Describe uncertain evidence of an advantage; do not search seeds. |
| A later-information run improves | Earliest versus direct antecedent rule | Label it as a different prediction task. |
Revise the model without training on the validation result
If prediction is weak, identify a substantive reason for revision: poorly measured inputs, missing information or unnecessary complexity. Keep the original result and label any search through alternatives as exploratory. Repeatedly trying models against the same validation results gradually uses those results as training information.
For a final predictive claim, use a fresh suitable assessment after model development. If cases are time ordered or clustered within classes, ordinary shuffled folds may not represent the intended future prediction problem. This independent synthetic exercise does not resolve those design choices for a real student study.
State what was known before stating how well you predicted
Methods outline: We predicted Intention and Use indicators from earliest-antecedent Belonging and Support information using [fold count] folds repeated [count] times with seed [value]. Preparation was learned from training cases. We compared PLS errors with IA and LM and read paired CVPAT results at [confidence level]. We retained [independent case count] and recorded [availability and failures].
Results outline: For [target], PLS RMSE was [value], compared with [IA] and [LM]. The aggregate PLS-minus-benchmark loss difference was [value and uncertainty] against IA and [value and uncertainty] against LM. [Explain disagreements across targets.] The Direct antecedents analysis used additional intention information and is reported as a separate task.
| Report | Include |
|---|---|
| Prediction question | Targets and information available at the intended decision time. |
| Validation plan | Folds, repetitions, seed, case selection and row order. |
| Preparation | Training-only transformations and missing-data handling. |
| Errors | Named indicator, loss measure and both available benchmarks. |
| CVPAT | Subtraction direction, uncertainty, target level and independent count. |
| Revisions | Exploratory changes and any fresh assessment used afterward. |
Write a prediction claim that states what was known
Using five folds repeated twice with earliest antecedents, the model produced lower held-out RMSE than both the training-mean and linear benchmarks for all six target indicators in the 250-case teaching file. Aggregate paired squared-error differences favoured PLS against both benchmarks.
Report the seed and fold plan, the information available at prediction time, all target errors and the unit used in the paired test. Keep the later direct-antecedents analysis separate.
Explain the predictions for this practice file
- Compare PLS, training-mean and linear-model RMSE for all six target indicators. State which method has the smaller error in each row.
- Explain why the aggregate PLS-minus-benchmark loss differences favour PLS in this file, using their intervals and the stated test direction.
- Distinguish the 250 independent students from the 500 held-out predictions per indicator.
- State why held-out Intention is unavailable in the earliest-antecedents task and available in the separate direct-antecedents task.
- Report the two prediction tasks separately. Keep conclusions limited to these synthetic data and the stated information rules.
- Optional contrast: explain how a model could improve on the mean benchmark while performing worse than the linear benchmark, using the separate hypothetical example.