ModEx

Model Examiner (Beta) - Return to DASH Lab

Copyright © 2022-2023 The Authors

Model Examiner (ModEx):

ModEx allows near real-time interaction with your model only using probabilistic sensitivity analyses (PSA) results from your model.

ModEx is based on peer-reviewed research on the use of regression metamodeling. In the background ModEx fits a regression metamodeling approach to the data. From this it allows visual and numerical displays, interactive incremental cost-effectiveness ratio (ICER) display, result decomposition into various parameters, and very fast EVPPI and EVSI computations. ModEx uses peer-reviewed methods, and Open Source R packages developed by the Decision Analysis in R for Technology and Health workgroup (DARTH).


To start exploring ModEx, please upload a PSA dataset in the next tab (Import files)


Here is a summary of the various tabs:

  1. Import files: you can upload a CSV of your PSA result using this tab and check the uploaded data.
  2. Diagnostics: shows the quality of the metamodel fit, and a check on pair-wise parameter correlations
  3. Sensitivity: numerical and graphical display of sensitivity analyses, including summaries, CEAC, Scatterplot, Regression summaries, and graphical displays of tornado, one-way and two-way plots.
  4. ICER: interactive display of the ICER under various parameter settings.
  5. Decomposition: interacive displays the contribution of each parameter to each outcomes
  6. Report: allows producing a report of all the analyses conducted and saving it locally
  7. EVSI: (coming soon!) computes the expected value of partial perfect Information (EVPPI) and expected value of sample information (EVSI) using extremely fast algorithms

Disclaimer: This application is provided AS-IS and comes with absolutely no warranty.

If you use ModEx please cite our papers

  1. Jalal, H., Dowd, B., Sainfort, F., & Kuntz, K. M. (2013). Linear regression metamodeling as a tool to summarize and present simulation model results. Medical Decision Making, 33(7), 880-890.
  2. Jalal, H., & Alarid-Escudero, F. (2018). A Gaussian approximation approach for value of information analysis. Medical Decision Making, 38(2), 174-188.

Setting up your files for import

Here you can import your probabilistic senesitivity analysis (PSA) dataset. The PSA dataset must follow a specific format.
Your PSA dataset must be uploaded as a single CSV file.
The PSA dataset must follow the following format: (Please download the example file below)
- Rows are PSA model runs, and columns are model parameters, and cost and effectivenss outcomes
- Columns indicating model input parameters must start with 'p_'.
- Columns indicating cost outcomes must start with 'c_'.
- Columns indicating effectiveness outcomes must start with 'e_'.
- There should be an equal number of cost and effectiveness columns.

First, define calculation settings

NHB = Net Health Benefit, NMB = Net Monetary Benefit.

Then, use an Example PSA dataset

You can start exploring ModEx using a PSA dataset from a stylistic Markov model involving cancer therapy. The model involves 8 input parameters and 3 treatment strategies. To learn more about this example, please refer to Jalal et al. 2013 MDM paper on the Home tab. You can also downlaod this dataset here.

Download PSA.csv

Or, import your own PSA dataset

Note: Once you upload the data, you cannot change the NB units or WTP. If you want to modify these settings, please do the changes first and then re-upload your PSA.

Uploaded Data Check

Parameters

Costs

Effects

Setting

Pairwise Correlation Analysis

This tab provides a visual and numerical analysis of the correlations between different input parameters. Scatterplots are provided for each pair of variables along with their correlation coefficients.

Collinear Variables


                      

                    

                  

                

              

            

          

          

            
Running diagnostics for the linear metamodel

      ModEx fits a model in the form Y = β*X + e, where X are the model input parameters, 
      Y are the model outputs (costs and QALYs), and β are the regression coefficients. 
      This tab shows the fits for each outcome and also the residuals to determine the degree of fit,
      and also guide if further modificaiton of the parameters are needed to improve the fit.
      

      

      The Correlation Analysis shows the level of correlation in the pair-wise parameters.  
      Parameters that are entirely computed from other parameters should be avoided.
      
      

            

              
              

                

                  

                    

                      
                    

                    

                      

                      

                    

                  

                

              

            

          

          

            
Sensitivity

      This tab produces both tabular and graphical sensitivity analyses results.
      

      

      Input/Output summary provide simple summaries to the model inputs and outputs, including the NB.
      

      

      CEAC and Scatterplot produces the indicated plots. 
      

      

      Regression produces the regression coefficients for the raw (unstandardized) and 
      standardized (normalized) inputs. 
      

      

      The Deterministic plots tab shows Tornado Diagram, One-way and Two-way sensitivity plots. 
      

      

      All using the same PSA dataset uploaded, and the lambda and NB computation type chosen on the Home tab.
      

            

              
              

                

                  
Parameter Summary

                  

                  
Cost Summary

                  

                  
Effect Summary

                  

                  
NB Summary

                  

                

                

                  

                    

                      

                        
                        

                          

                            
                          

                        

                      

                    

                    

                      

                        
                        

                          

                            
                          

                        

                      

                    

                  

                  

                    

                      

                      

                      

                    

                    

                      

                    

                  

                

                

                  

                    

                      

                        
                        

                          

                            
                          

                          

                            
                          

                        

                      

                    

                    

                      
Filter

                      

                        

                          
                        

                      

                      

                        

                          
                        

                      

                      

                        

                          
                        

                      

                    

                  

                  
R-squared Value

                  

                  
Unstandardized Regression Results

                  
The coefficients indicate change in the outcome due to 1 unit change in each parameter. 
                         The intercept indicates the outcome value if all parameters were = 0.

                  

                  
Standardized Regression Results

                  
The coefficients indicate change in the outcome due to 1 standard deviation change in each parameter. 
                         The intercept indicates the average outcome value if all parameters were = their mean values.

                  

                

                

                  

                    
                  

                  

                    

                    

                    

                  

                

              

            

          

          

            
ICER

              Here you can interact with the Incremental Cost-Effectiveness Ratio (ICER) plot by adjusting the model parameters.
              It does this quickly by using the regression metamodel in the background.
              

            

              

                
              

              

                

                

              

            

          

          

            
Decomposition

      Here, you can reveal how much each parameter contributes to each strategy outcome, and how these parameters' 
      contributions change as you interactively adjusts the model parameters. 
      

            

              

                
              

              

                

              

            

          

          

            
Download summary report

            
This report will contain all the tables and figures generated from your uploaded dataset. Please be patient it may take some time to produce.

            

              
              

                
                
                
              

            

            
              
              Download summary report
            
            

            

          

          

            
About us

            
This web tool is an 
              R Shiny Server application.

            
It developed at the Decision Analysis, Simulation and Health economics (DASH) Lab at University of Ottawa's School of Epidemiology and Public Health
       School of Epidemiology and Public Health by Hawre Jalal and Minjie Cao, and 
              supported by the Canada Research Chair in Health Economics.

            
The regression metamodeling method was developed by Hawre Jalal in collaboration with 
       Karen Kuntz, Bryan Dowd, Francois Sainfort and John Nyman and was partly 
              supported by Hawre Jalal's AHRQ Dissertation Award R36HS020868

            
The Gaussian approximation method was developed by Hawre Jalal in collaboration with Fernando Alarid-Escudero 
              and was partly supported by Hawre Jalal's Career Development Award K01DA048985.

            
If you use ModEx please cite our papers

            
    
              
  1. Jalal, H., Dowd, B., Sainfort, F., & Kuntz, K. M. (2013). Linear regression metamodeling as a tool to summarize and present simulation model results. Medical Decision Making, 33(7), 880-890.
    2. 
              
  2. Jalal, H., & Alarid-Escudero, F. (2018). A Gaussian approximation approach for value of information analysis. Medical Decision Making, 38(2), 174-188.
    3. 
            

            
The initial version of ModEx used the Shiny App code from the Sheffield Accelerated Value of Information (SAVI). 
              Since then, it got a full re-write of the code to reveal health economic model characteristics using metamodelling. 
              In addition, ModEx allows to interactively allow flexible and extremely fast EVSI computation.

            
If you have any questions, please contact us at hjalal@uottawa.ca.