#### Model Examiner (Beta) - Return to DASH Lab

###### Copyright © 2022-2023 The Authors

### Model Examiner (ModEx):

ModEx allows near real-time interaction with your model only using probabilistic sensitivity analyses (PSA) results from your model.

ModEx is based on peer-reviewed research on the use of regression metamodeling. In the background ModEx fits a regression metamodeling approach to the data. From this it allows visual and numerical displays, interactive incremental cost-effectiveness ratio (ICER) display, result decomposition into various parameters, and very fast EVPPI and EVSI computations. ModEx uses peer-reviewed methods, and Open Source R packages developed by the Decision Analysis in R for Technology and Health workgroup (DARTH).

**To start exploring ModEx, please upload a PSA dataset in the next tab (Import files)**

Here is a summary of the various tabs:

- Import files: you can upload a CSV of your PSA result using this tab and check the uploaded data.
- Diagnostics: shows the quality of the metamodel fit, and a check on pair-wise parameter correlations
- Sensitivity: numerical and graphical display of sensitivity analyses, including summaries, CEAC, Scatterplot, Regression summaries, and graphical displays of tornado, one-way and two-way plots.
- ICER: interactive display of the ICER under various parameter settings.
- Decomposition: interacive displays the contribution of each parameter to each outcomes
- Report: allows producing a report of all the analyses conducted and saving it locally
- EVSI: (coming soon!) computes the expected value of partial perfect Information (EVPPI) and expected value of sample information (EVSI) using extremely fast algorithms

Disclaimer: This application is provided AS-IS and comes with absolutely no warranty.

**If you use ModEx please cite our papers**

- Jalal, H., Dowd, B., Sainfort, F., & Kuntz, K. M. (2013). Linear regression metamodeling as a tool to summarize and present simulation model results. Medical Decision Making, 33(7), 880-890.
- Jalal, H., & Alarid-Escudero, F. (2018). A Gaussian approximation approach for value of information analysis. Medical Decision Making, 38(2), 174-188.

#### Setting up your files for import

Here you can import your probabilistic senesitivity analysis (PSA) dataset. The PSA dataset must follow a specific format.Your PSA dataset must be uploaded as a single CSV file.

The PSA dataset must follow the following format: (Please download the example file below)

- Rows are PSA model runs, and columns are model parameters, and cost and effectivenss outcomes

- Columns indicating model input parameters must start with 'p_'.

- Columns indicating cost outcomes must start with 'c_'.

- Columns indicating effectiveness outcomes must start with 'e_'.

- There should be an equal number of cost and effectiveness columns.

#### First, define calculation settings

NHB = Net Health Benefit, NMB = Net Monetary Benefit.

#### Then, use an Example PSA dataset

You can start exploring ModEx using a PSA dataset from a stylistic Markov model involving cancer therapy. The model involves 8 input parameters and 3 treatment strategies. To learn more about this example, please refer to Jalal et al. 2013 MDM paper on the Home tab. You can also downlaod this dataset here.

#### Or, import your own PSA dataset

Note: Once you upload the data, you cannot change the NB units or WTP. If you want to modify these settings, please do the changes first and then re-upload your PSA.

### Uploaded Data Check

#### Parameters

#### Costs

#### Effects

#### Setting

#### Pairwise Correlation Analysis

This tab provides a visual and numerical analysis of the correlations between different input parameters. Scatterplots are provided for each pair of variables along with their correlation coefficients.#### Collinear Variables

#### Running diagnostics for the linear metamodel

ModEx fits a model in the form Y = β*X + e, where X are the model input parameters, Y are the model outputs (costs and QALYs), and β are the regression coefficients. This tab shows the fits for each outcome and also the residuals to determine the degree of fit, and also guide if further modificaiton of the parameters are needed to improve the fit.The Correlation Analysis shows the level of correlation in the pair-wise parameters. Parameters that are entirely computed from other parameters should be avoided.

#### Sensitivity

This tab produces both tabular and graphical sensitivity analyses results.Input/Output summary provide simple summaries to the model inputs and outputs, including the NB.

CEAC and Scatterplot produces the indicated plots.

Regression produces the regression coefficients for the raw (unstandardized) and standardized (normalized) inputs.

The Deterministic plots tab shows Tornado Diagram, One-way and Two-way sensitivity plots.

All using the same PSA dataset uploaded, and the lambda and NB computation type chosen on the Home tab.

#### Parameter Summary

#### Cost Summary

#### Effect Summary

#### NB Summary

#### Filter

#### R-squared Value

#### Unstandardized Regression Results

###### The coefficients indicate change in the outcome due to 1 unit change in each parameter. The intercept indicates the outcome value if all parameters were = 0.

#### Standardized Regression Results

###### The coefficients indicate change in the outcome due to 1 standard deviation change in each parameter. The intercept indicates the average outcome value if all parameters were = their mean values.

#### ICER

Here you can interact with the Incremental Cost-Effectiveness Ratio (ICER) plot by adjusting the model parameters. It does this quickly by using the regression metamodel in the background.#### Decomposition

Here, you can reveal how much each parameter contributes to each strategy outcome, and how these parameters' contributions change as you interactively adjusts the model parameters.### Download summary report

This report will contain all the tables and figures generated from your uploaded dataset. Please be patient it may take some time to produce.

### About us

This web tool is an R Shiny Server application.

It developed at the Decision Analysis, Simulation and Health economics (DASH) Lab at University of Ottawa's School of Epidemiology and Public Health School of Epidemiology and Public Health by Hawre Jalal and Minjie Cao, and supported by the Canada Research Chair in Health Economics.

The regression metamodeling method was developed by Hawre Jalal in collaboration with Karen Kuntz, Bryan Dowd, Francois Sainfort and John Nyman and was partly supported by Hawre Jalal's AHRQ Dissertation Award R36HS020868

The Gaussian approximation method was developed by Hawre Jalal in collaboration with Fernando Alarid-Escudero and was partly supported by Hawre Jalal's Career Development Award K01DA048985.

**If you use ModEx please cite our papers**

- Jalal, H., Dowd, B., Sainfort, F., & Kuntz, K. M. (2013). Linear regression metamodeling as a tool to summarize and present simulation model results. Medical Decision Making, 33(7), 880-890.
- Jalal, H., & Alarid-Escudero, F. (2018). A Gaussian approximation approach for value of information analysis. Medical Decision Making, 38(2), 174-188.

The initial version of ModEx used the Shiny App code from the Sheffield Accelerated Value of Information (SAVI). Since then, it got a full re-write of the code to reveal health economic model characteristics using metamodelling. In addition, ModEx allows to interactively allow flexible and extremely fast EVSI computation.

If you have any questions, please contact us at hjalal@uottawa.ca.