text:
## Abstract

Rupture risk assessment of abdominal aortic aneurysms (AAAs) by means of quantifying wall stress is a common biomechanical strategy. However, the clinical translation of this approach has been greatly limited due to the complexity associated with the computational tools required for its implementation. Thus, being able to estimate wall stress using nonbiomechanical markers that can be quantified as a direct outcome of clinical image segmentation would be advantageous in improving the potential implementation of said strategy. In the present work, we investigated the use of geometric indices to predict patient-specific AAA wall stress by means of a novel neural network (NN) modeling approach. We conducted a retrospective review of existing clinical images of two patient groups: 98 asymptomatic and 50 symptomatic AAAs. The images were subject to a protocol consisting of image segmentation, processing, volume meshing, finite element modeling, and geometry quantification, from which 53 geometric indices and the spatially averaged wall stress (SAWS) were calculated. SAWS estimated from finite element analysis was considered the gold standard for the predictions. We developed feed-forward NN models composed of an input layer, two dense layers, and an output layer using Keras, a deep learning library in python. The NN models were trained, tested, and validated independently for both AAA groups using all geometric indices, as well as a reduced set of indices resulting from a variable reduction procedure. We compared the performance of the NN models with two standard machine learning algorithms (MARS: multivariate adaptive regression splines and GAM: generalized additive model) and a linear regression model (GLM: generalized linear model). With the reduced sets of indices, the NN-based approach exhibited the highest mean goodness-of-fit (for the symptomatic group 0.71 and for the asymptomatic group 0.79) and lowest mean relative error (17% for both groups). In contrast, MARS yielded a mean goodness-of-fit of 0.59 for the symptomatic group and 0.77 for the asymptomatic group, with relative errors of 17% for the symptomatic group and 22% for the asymptomatic group. GAM had a mean goodness-of-fit of 0.70 for the symptomatic group and 0.80 for the asymptomatic group, with relative errors of 16% for the symptomatic group and 20% for the asymptomatic group. GLM did not perform as well as the other algorithms, with a mean goodness-of-fit of 0.53 for the symptomatic group and 0.70 for the asymptomatic group, with relative errors of 19% for the symptomatic group and 23% for the asymptomatic group. Nevertheless, the NN models required a reduced set of 15 and 13 geometric indices to predict SAWS for the symptomatic and asymptomatic AAA groups, respectively. This was in contrast to the reduced set of nine and eight geometric indices required to predict SAWS with the MARS and GAM algorithms for each AAA group, respectively. The use of NN modeling represents a promising alternative methodology for the estimation of AAA wall stress using geometric indices as surrogates, in lieu of finite element modeling. The performance metrics of NN models are expected to improve with significantly larger group sizes, given the suitability of NN modeling for “big data” applications.

## 1 Introduction

Abdominal aortic aneurysm (AAA) is a life-threatening condition where a high number of patients with a ruptured AAA die before clinical intervention is possible [1]. In current clinical practice, maximum AAA diameter (Dmax) and aneurysm growth rate are the most widely utilized clinical markers for assessing progression of the disease [2]. Ruptured AAAs are the 13th leading cause of death in the United States with an associated mortality rate of 90% while causing nearly 11,000 deaths per year [3]. Gender is also a well-established risk factor for the development of an AAA, with a 4:1 male to female ratio [4]. The gold standard for recommending elective repair for AAA patients is a maximum diameter of 5.5 cm in men and 5.0 cm in women [3]. However, progressive dilation weakens the aortic wall, decreasing its ability to withstand blood pressure, and ultimately leading to AAA rupture. The latter is a mechanical event that occurs when the local wall stress exceeds the local wall strength. Based on this principle, several biomechanical markers such as peak wall stress (PWS), 99th percentile wall stress (99thWS), spatially averaged wall stress (SAWS), and rupture potential index have been postulated as superior predictors of AAA rupture compared to Dmax alone [516]. As patient-specific wall stress calculations require prior knowledge of individual AAA material properties, geometric surrogates such as wall thickness, surface curvature, size, and shape measures [14,1621] have been postulated as image-based predictors of wall stress. Such geometric indices could be used as part of a computational tool for AAA rupture risk assessment in lieu of complex finite element analysis (FEA) [14,22].

The overall objective of the present work is to investigate the potential of geometric indices to predict patient-specific AAA wall stress by means of a novel neural network (NN)-based modeling approach. NNs are a set of algorithms designed to recognize patterns in large sets of data. Previously, NN-based prediction of aortic wall stress (using geometric indices and other similar variables) has been proposed in computational studies of idealized vascular geometries [2225]. Our goals were (1) to develop a NN-based predictive model using geometric indices as predictors and SAWS as the response; (2) investigate the prediction accuracy when using a reduced set of predictors; and (3) compare the prediction accuracy of the NN with standard machine learning (ML) algorithms.

## 2 Materials and Methods

### 2.1 Human Subjects Data.

The source of data for this study consisted of two patient groups: 50 symptomatic AAA patients who underwent an emergent aneurysm repair and 98 asymptomatic AAA patients who underwent an elective aneurysm repair. For each patient, we acquired retrospectively the last abdominal computed tomography angiography (CTA) scan available prior to the emergent or elective repair. A flowchart illustrating the protocol followed in the study is shown in Fig. 1. The human subjects' research protocol was approved by the Institutional Review Boards at Allegheny General Hospital (Pittsburgh, PA) and Northwest Memorial Hospital (Chicago, IL). Informed consent was not required as this was a retrospective review of existing, de-identified medical records.

Fig. 1
Fig. 1

### 2.2 Three-Dimensional Image Reconstruction.

CTA images (slice thickness 1.0–3.0 mm; 512 × 512 pixels) ranging from immediately distal to the left renal artery to approximately 5 cm distal to the iliac bifurcation were used in standard Digital Imaging and Communications in Medicine (DICOM) format. These DICOM files were utilized as inputs for the in-house matlab-based (Mathworks Inc., Natick, MA) segmentation script AAAVasc (v1.0.3, The University of Texas at San Antonio, San Antonio, TX) [26]. AAAVasc segments the aorta using three distinct boundaries: lumen, inner wall, and outer wall. Lumen segmentation was performed by means of a region-growing algorithm where the user selects a seed point (inside the lumen) that acts as an initiation point. The algorithm then searches for neighboring pixels in eight directions from this seed pixel (top, bottom, left, right, and the four diagonals). The search is discontinued when a significant change in intensity is achieved (measured using image pixel intensity gradient), which signifies the end of the lumen region. This process was repeated for each image in the patient-specific set of CTA images. Next, the outer wall segmentation was performed using contour detection [27]. AAAVasc provides the user with possible outer wall contours and the user selects the best-fitted outer wall. To segment the inner wall, a simple feed forward neural network (FFNN)1 was utilized. In this step, the user selects approximately five to ten seed points that could be part of the inner wall boundary and seed points that are not part of the wall region. The FFNN was subsequently trained and all the pixels between the lumen and the outer wall were labeled as either (a) part of the wall or (b) not part of the wall. Using the aforementioned labels, the FFNN then allocated the inner wall boundary for each image. With the three boundaries identified, the three regions (lumen, intraluminal thrombus (ILT), and wall) were subsequently exported as binary masks and as a point cloud for further processing. The ILT region was represented in each binary mask by the two-dimensional region between the inner wall and lumen boundaries.

### 2.3 Computational Analysis of Patient-Specific AAA Wall Stress.

Using another custom matlab-based meshing script, AAAMesh [28,29], a quadratic hexahedral mesh, is created from the aforementioned binary masks for each AAA geometry. The volumetric mesh has a patient-specific wall thickness distribution estimated during image segmentation, similar to previously reported studies [5,14,16,30]. ILT was not meshed in the model, while the wall mesh density was fixed to account for two-element thick wall. The resulting mesh consisted of approximately 60,000–100,000 elements, which were exported in a nastran file format and processed with the finite element solver adina (Adina R&D Inc., Watertown, MA) for subsequent wall stress analysis. We previously performed a mesh sensitivity analysis for quasi-static AAA FEA modeling [27]. The patient-specific FEA simulations were performed in 24 time steps with an intraluminal pressure of 120 mmHg and the degrees-of-freedom at the proximal and distal ends fixed in all directions and rotations. A generalized neo-Hookean material model was used for the AAA wall, with its strain energy function given by Eq. (1)
$W=c1(I1−3)+c3(I1−3)2$
(1)

where $W$ is the strain energy density, $I1$ is the second invariant of the Cauchy–Green tensor, $c1$ and $c3$ are constitutive parameters evaluated experimentally, and the Poison's ratio was taken to represent a nearly incompressible material ($ν=0.49$) [31,32]. The parameters ($c1=17.4 N/cm2$ and $c3=188.1 N/cm2$) were adopted from the study by Raghavan and Vorp [31], which involved uni-axial tensile testing of AAA wall tissue specimens from 69 patients. From the FEA simulations, first principal stresses were obtained and PWS, 99thWS, and SAWS were estimated using ensight (Ansys Inc., Canonsburg, PA). Previously, SAWS was reported to have stronger correlations with aneurysmal geometric indices, compared to PWS and 99thWS [14,16]. These global stress measures are defined in Appendix B of the Supplementary Material on the ASME Digital Collection.

### 2.4 Quantification of Geometric Indices.

Fifty-three geometric indices that best describe the overall shape, size, and curvature of the AAA models were estimated using another in-house matlab script developed by Shum [26] and Shum et al. [33,34] and also reported in [14,16]. The patient-specific binary masks and point clouds bounding the three regions (lumen, ILT, and wall) were utilized to estimate one-dimensional (such as average wall thickness (THave) and max diameter (Dmax)) and two-dimensional geometric indices (e.g., wall surface area (S)). The volumetric meshes were utilized to evaluate three-dimensional geometric indices such as AAA sac volume (V). The Biquintic Hermite finite element method, implemented from Lee et al. [35], was used to evaluate global curvature-based indices (such as area-averaged mean curvature (MAA)). Local curvature distributions were utilized to estimate the global curvature indices using a high-order interpolation scheme. Local principal curvatures $k1$ and $k2$ are estimated using Eqs. (2) and (3)
$k1=a+c+(a−c)2+4b2$
(2)
$k2=a+c−(a−c)2+4b2$
(3)

where $a$, $b$, and $c$ are constants evaluated at every outer wall surface node. The detailed formulations and abbreviations of all geometric indices are provided in Appendix A of the Supplementary Material on the ASME Digital Collection.

### 2.5 Neural Network Architecture and Machine Learning Algorithms.

A feed-forward NN was developed consisting of an input layer, two dense layers, and an output layer (as shown in Fig. 2) using Keras, a deep learning library in python (Python Software Foundation, Wilmington, DE). These layers were connected using appropriate activation functions where the resultant output layer was the weighted sum of the input nodes. Four NN models were developed for both the symptomatic and asymptomatic AAA patient groups using several combination of layers and activation functions. The models were then evaluated for various epochs and the best prediction model for both groups was reported. Mean square error (MSE) and mean average error (MAE) were used to estimate the accuracy of the models. Further, we also compared these NN models with ML algorithms (specifically, generalized additive model (GAM) [36] and multivariate adaptive regression splines (MARS) [37]) applied to the same groups, following a ML protocol reported previously in Ref. [16].

Fig. 2
Fig. 2

### 2.6 Training, Testing, and Validation Datasets.

The data were randomly divided into training, testing, and validation datasets (see schematic of study design in Fig. 3). For the symptomatic group, the NN model was trained and tested on 40 AAAs with a 70:30 split. This trained symptomatic NN model was then validated using 10 AAAs that were not part of the training or testing datasets. Similarly, for the asymptomatic group, the NN was trained and tested on 80 AAAs with a 70:30 split, and the trained model was further validated using 18 AAAs (that were not part of the training or testing datasets). The aforementioned datasets were also utilized to train and validate the ML algorithms (GAM and MARS).

Fig. 3
Fig. 3

### 2.7 Variable Reduction Procedure.

As shown in Fig. 3, two different sets of input variables were evaluated for each of the NN models for both AAA groups (symptomatic and asymptomatic). The first variable set consisted of all 53 geometric indices as predictors and SAWS as the response. The second variable set consisted of a reduced set of variables (selected using either Pearson's correlation analysis or MARS) as predictors and SAWS as the response. The variable reduction was performed using Pearson's correlation in conjunction with Bonferroni correction (to reduce the probability of a type I error while performing multiple hypotheses tests) [14,16,38]. To identify the geometric indices that have the best correlation with SAWS, a series of tests of hypothesis was carried out. Specifically, for each $j=1, …, m (where m=53)$, the null hypothesis $H0(j): ρj=0$ was tested against the alternative hypothesis $Ha(j): ρj≠0$, where $ρj$ is the population correlation coefficient between the jth geometric index and SAWS. For each $j, H0(j)$ is rejected for large values of $Tj$ according to Eq. (4)
$Tj= rjn−21−r j2$
(4)
for which $n$ is the sample size and $rj$ is determined by Eq. (5)
$rj= ∑i=1n(xij− xj¯)(yi− y ¯) (∑i=1n(xij− xj¯)2.∑i=1n(yi− y ¯)2)$
(5)

where $rj$ is the sample (Pearson's) correlation coefficient between the jth geometric index and SAWS, $xij$ is the value of the jth geometric index from the ith patient; $xj¯=1n∑1nxij$, $yi$ is SAWS for the $ith$ patient, and $y¯=1n∑1nyi$. The p-values of these tests are computed with the fact that $Tj$ has a t-distribution with $n−2$ degrees-of-freedom when $H0(j)$ is true [14,16]. Since there are 53 geometric indices, the adjusted level of significance,$αc,$ was set to $0.05/53=0.00094$ (i.e., the Bonferroni correction). Geometric indices that exhibited no significant correlations with SAWS were removed from the set of reduced geometric indices. Further, collinear indices, i.e., indices with a $ρ$ value > 0.98 among each other, were also removed. After removal of the nonsignificant variables, the reduced subset of geometric indices was utilized as input neurons for the NN models.

For the ML algorithms, MARS was utilized to refine the best set of geometric indices for predicting SAWS in both MARS and GAM. We also compared the prediction accuracy of MARS and GAM with all 53 indices as inputs. The correlation analysis and variable selection procedures were performed with the R [39] packages corrgram [40], earth [37], and gam [41], respectively. Additional information on how MARS performs the variable refinement protocol for the ML algorithms is provided in Appendix C of the Supplementary Material on the ASME Digital Collection.

### 2.8 Neural Network Optimization.

The performance of the NN was evaluated using a loss function, which is calculated by matching the target/response value (SAWS estimated from FEA) with the NN-predicted value (SAWS predicted by the geometric indices). The weights of each predictor are updated for each epoch to minimize the loss function. Since the NN was used to predict a continuous variable, the loss functions were MAE and MSE, calculated by Eqs. (6) and (7)
$MAE= 1n∑k=1n|zk− ẑk|$
(6)
$MSE= 1n∑k=1n(zk−ẑk)2$
(7)

where $n$ is the sample size, $zk$ is the actual value (the FEA-estimated SAWS), and $ẑk$ is the predicted value (the NN-estimated SAWS).

When training the NN models, “early stopping” was used to quantify the optimum epoch for each model. Early stopping refers to the methods that allow specifying the performance measure to monitor the trigger, and once triggered, it will stop the training process. MAE was used to monitor the trigger and stop the training process. When the validation loss starts to increase in the next few epochs after the lowest MAE value has been achieved, early stopping will trigger the NN models to stop training, thereby eliminating over-fitting.

### 2.9 Comparison of Prediction Accuracies.

The prediction accuracies of NN, GAM, and MARS were compared using two metrics: goodness-of-fit and percentage relative error. The goodness-of-fit value (R2) for all the computational algorithms was obtained by performing a regression analysis of the predicted and estimated SAWS derived from the validation dataset. The relative error was calculated according to Eq. (8)
$Relative error %=( zk− ẑkẑk) × 100$
(8)

The aforementioned metrics were reported for the validation dataset, i.e., ten AAAs in the symptomatic group and 18 AAAs in the asymptomatic group, respectively. Both metrics were evaluated for all three algorithms (NN, GAM, and MARS) using all 53 indices and their respective reduced sets of indices.

### 2.10 Linear Regression.

Using the geometric indices selected from the Pearson's correlation with Bonferroni correction, a linear regression model was fit using the “stats” package in RStudio [42]. A generalized linear model (GLM) was fit for the asymptomatic group using the reduced geometric indices as predictors and SAWS as the response. Similarly, GLM was also used for the symptomatic group with the reduced geometric indices as predictors and SAWS as the response. For the asymptomatic group, GLM was fit using 80 AAAs and tested on the remaining 18 AAAs (that were not part of the fit); for the symptomatic group, GLM was fit using 40 AAAs and tested using the remaining 10 AAAs.

## 3 Results

### 3.1 Wall Stress.

Spatially averaged wall stress, estimated from patient-specific FEA, was higher for the symptomatic group compared to the asymptomatic group. The mean SAWS for the symptomatic and asymptomatic groups was 30.1±11.4 N/cm2 and 22.3±8.3 N/cm2, respectively. The mean values and standard deviations of PWS, 99thWS, and SAWS are included in Appendix D of the Supplementary Material on the ASME Digital Collection.

### 3.2 Geometric Indices.

The mean values and standard deviations of all geometric indices are included in Appendix D of the Supplementary Material. The NN models used these indices as predictors and SAWS as the response, with one model for the symptomatic group and another model for the asymptomatic group.

### 3.3 Neural Network Models

#### 3.3.1 Reduced Subset of Geometric Indices for the Neural Network.

For the NN models, Pearson's correlations were performed to reduce the number of redundant and nonsignificant geometric indices. Figures 4 and 5 show the correlation matrices representing all variables that correlate most with SAWS for the symptomatic AAA group (n = 98) and the asymptomatic AAA group (n = 50), respectively. The correlation coefficients between any two variables are represented by the size and color of the dot in the matrices. The reduced list of indices from the correlation analysis for both AAA groups are summarized in Table 1. Of the 53 indices, 15 and 13 indices were significantly correlated (p-value < 0.00094) with SAWS for the symptomatic and asymptomatic groups, respectively. Of the aforementioned reduced sets of indices, S, V, Dmax, Dave, THave, THDmax, THmedian, and dcmax were common to both groups. For the symptomatic group, THDmax exhibited the highest negative association with SAWS ($ρ=0.652$, p-value ≪ 0.05), whereas VILT exhibited the highest positive association with SAWS ($ρ=0.494$, p-value ≪ 0.05). Similarly, for the asymptomatic group, S and MAA exhibited the strongest positive and negative correlations with SAWS, respectively ($ρ=0.665$, p-value ≪ 0.05 versus $ρ=0.555$, p-value  ≪ 0.05).

Fig. 4
Fig. 4
Fig. 5
Fig. 5
Table 1

Pearson's correlation analysis of SAWS with geometric indices for the symptomatic (n = 50) and asymptomatic (n = 98) AAA groups

SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
Geometric indicesPearson's correlation coefficient $(ρ)$p-ValuePearson's correlation coefficient $(ρ)$p-Value
Dmax0.441≪0.050.596≪0.05
S0.472≪0.050.665≪0.05
V0.416≪0.050.654≪0.05
IPR–0.332≪0.05
Dmaxdir0.427≪0.05
Dave0.433≪0.050.598≪0.05
H0.415≪0.05
MAA–0.555≪0.05
L0.400≪0.050.403≪0.05
VILT0.494≪0.05
$γ$0.416≪0.05
TTmax0.327≪0.05
dcmax0.320≪0.050.393≪0.05
dc0.346≪0.05
THmode–0.560≪0.05
THDmax–0.652≪0.05–0.389≪0.05
THmedian–0.588≪0.05–0.331≪0.05
THmin–0.335≪0.05
THave–0.593≪0.05–0.353≪0.05
SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
Geometric indicesPearson's correlation coefficient $(ρ)$p-ValuePearson's correlation coefficient $(ρ)$p-Value
Dmax0.441≪0.050.596≪0.05
S0.472≪0.050.665≪0.05
V0.416≪0.050.654≪0.05
IPR–0.332≪0.05
Dmaxdir0.427≪0.05
Dave0.433≪0.050.598≪0.05
H0.415≪0.05
MAA–0.555≪0.05
L0.400≪0.050.403≪0.05
VILT0.494≪0.05
$γ$0.416≪0.05
TTmax0.327≪0.05
dcmax0.320≪0.050.393≪0.05
dc0.346≪0.05
THmode–0.560≪0.05
THDmax–0.652≪0.05–0.389≪0.05
THmedian–0.588≪0.05–0.331≪0.05
THmin–0.335≪0.05
THave–0.593≪0.05–0.353≪0.05

Correlation coefficients $(ρ)$ > |0.5| are highlighted in bold-faced text.

#### 3.3.2 Training and Validation Loss for the NN Models.

Detailed training and validation loss characteristics for the four NN models are shown in Figs. 6(a)6(d), which illustrates the convergence of training and validation losses over the epoch for each of the models. Training loss is defined as the overall error on the training set of the data, whereas validation loss is the overall error of the validation dataset through the trained network. Training and validation losses were estimated using MSE. Figure 6(c) shows the convergence of MSE for the training and validation data sets using the reduced set of geometric indices for the symptomatic group. The training MSE at 0 epoch was 0.9 and at 100 epochs was 0.18; similarly, the validation MSE at 0 epoch was 0.2, which converged at 100 epochs to 0.02. A similar trend can be seen for Fig. 6(d) for the asymptomatic group. Figures 6(a) and 6(b) show the training and validation losses for the symptomatic and asymptomatic groups, respectively, using all 53 geometric indices. The training and validation loss curves do not follow the trends seen in Figs. 6(c) and 6(d), which is because many indices (among the 53 indices) do not contribute toward the prediction of SAWS.

Fig. 6
Fig. 6

The training and testing MAE and MSE for the models are reported in Table 2. For the symptomatic AAA group, the training MAE and MSE for the model with 53 indices converged to 0.48 and 0.3, respectively, while the testing MAE and MSE for the model with 53 indices converged to 0.36 and 0.18, respectively. The training MAE and MSE for the model with 15 indices converged to 0.05 and 0.18, respectively, while the testing MSE and MAE for the model with 15 indices converged to 0.02 and 0.11, respectively. Since the MAE and MSE of the training and testing sets for the model using 53 indices was nearly two to six times higher than the model using 15 indices, we infer that a reduced set of 15 geometric indices highly correlated with SAWS is adequate to build the NN model for symptomatic AAA.

Table 2

Accuracy of the NN models with training (80%) and testing (20%) datasets for the symptomatic (n = 50) and asymptomatic (n = 98) AAA groups, expressed as a function of the MAE and MSE

SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
Dataset typeNumber of geometric indicesMAEMSENumber of geometric indicesMAEMSE
Training530.480.30530.140.03
Testing530.360.18530.270.10
Training150.180.05130.130.02
Testing150.110.02130.170.04
SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
Dataset typeNumber of geometric indicesMAEMSENumber of geometric indicesMAEMSE
Training530.480.30530.140.03
Testing530.360.18530.270.10
Training150.180.05130.130.02
Testing150.110.02130.170.04

The NN model metrics for the asymptomatic group are also reported in Table 2. For this group, the training MAE and MSE for the model with 53 indices were 0.14 and 0.03, respectively. The testing MAE and MSE were 0.27 and 0.10, respectively. Using a reduced set of 13 indices, the training MAE (0.13) and MSE (0.02) were slightly lower than the model with 53 indices, and the testing MAE (0.17) and MSE (0.04) were also lower than the model with 53 indices. Therefore, we infer that a reduced set of 13 geometric indices highly correlated with SAWS is adequate to build the NN model for asymptomatic AAA.

#### 3.3.3 Neural Network Outputs.

The results of the NN model predictions are shown in Figs. 7(a)7(d) and Table 3, where a comparison is shown of the prediction accuracies of the models for both groups. For the symptomatic group, the NN model based on 15 indices was trained for 70 epochs and predicted SAWS with a goodness-of-fit (R2) of 0.71. The average relative error of the predicted SAWS was 17%. Using all 53 indices, the NN was trained for 140 epochs. The goodness-of-fit (0.71) of the predicted SAWS and the average relative error (16%) of the prediction was similar to the NN model using 15 indices. For the asymptomatic group, using the 13 indices as input nodes, the NN trained for 150 epochs predicted SAWS with a goodness-of-fit of 0.79. The average relative error of the predicted SAWS for this network was 17%. Conversely, The NN model with 53 indices as input nodes was found to have a goodness-of-fit of 0.70 with the average relative error being 27%. The latter NN model was trained for 100 epochs.

Fig. 7
Fig. 7
Table 3

Comparison of prediction accuracies for all the algorithms for the symptomatic (n = 50) and asymptomatic (n = 98) AAA groups

SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
AlgorithmsNumber of geometric indicesGoodness of fit (R2)Relative error (%)Number of geometric indicesGoodness of fit (R2)Relative error (%)
NN530.7115.7530.7026.8
NN150.7116.8130.7917.4
MARS530.6513.5530.7722.5
MARS90.5917.480.7721.7
GAM531.01 × 10–5239.7530.0438.4
GAM90.7015.880.8020.0
GLM53ab53ab
GLM150.5318.5130.7022.6
SAWS (symptomatic AAA)SAWS (asymptomatic AAA)
AlgorithmsNumber of geometric indicesGoodness of fit (R2)Relative error (%)Number of geometric indicesGoodness of fit (R2)Relative error (%)
NN530.7115.7530.7026.8
NN150.7116.8130.7917.4
MARS530.6513.5530.7722.5
MARS90.5917.480.7721.7
GAM531.01 × 10–5239.7530.0438.4
GAM90.7015.880.8020.0
GLM53ab53ab
GLM150.5318.5130.7022.6
a

Model did not converge.

b

Relative error could not be estimated.

### 3.4 Machine Learning Models

#### 3.4.1 Reduced Subset of Geometric Indices for MARS and GAM.

The geometric indices utilized as input for the MARS and GAM algorithms were selected based on the variable reduction method built in MARS. This method chooses the optimal set of variables that contribute primarily toward a multivariable prediction model. From the application of such method, the geometric indices were reduced from 53 to nine for the symptomatic group (L, THmin, TTminLoc, GLN, VILT, Lsac, Dneck,p, BL, and THDmax) and from 53 to eight for the asymptomatic group (H, TTmaxLoc, S, V, IPR, THave, THmin, and THmod). GLN, VILT and THDmax were common to the reduced indices sets for the NN models and ML algorithms in the symptomatic group. Conversely, S, V, IPR, THave, and THmin were prevalent among the reduced indices for the asymptomatic group for both types of prediction algorithms.

#### 3.4.2 Machine Learning (MARS and GAM) Outputs.

The results of the MARS and GAM prediction analyses are shown in Figs. 8(a)8(d), 9(a)9(d), and Table 3. For the symptomatic group, SAWS predicted by the MARS algorithm using nine indices had a goodness-of-fit of 0.59 with an average relative error of 17%. The prediction from the GAM algorithm using the same nine indices exhibited a goodness-of-fit of 0.70 and an average relative error of 16% (compared to the FEA-estimated SAWS, which is considered to be the ground truth). SAWS predicted for the asymptomatic group using MARS with eight indices exhibited a goodness-of-fit of 0.77 and an average relative error of 22%. The prediction with GAM using the same eight indices exhibited a goodness-of-fit of 0.80 and an average relative error of 20% (compared to the FEA-estimated SAWS).

Fig. 8
Fig. 8
Fig. 9
Fig. 9

### 3.5 Linear Regression

#### 3.5.1 Reduced Subset of Geometric Indices for GLM

The correlation matrices illustrated in Figs. 4 and 5 were used to select indices that are significantly correlated to SAWS for both the symptomatic and asymptomatic groups. Hence, the same geometric indices used for the NN models were also used for the predictions with GLM, as 15 indices were found to be highly correlated with SAWS for the symptomatic group while 13 indices were found to be highly correlated with SAWS for the asymptomatic group (see Table 1).

#### 3.5.2 Linear Regression (GLM) Outputs.

The results of the GLM prediction analyses are shown in Figs. 8(e), 9(e), and Table 3. SAWS predicted by GLM for the symptomatic group using the reduced set of 15 indices exhibited a goodness-of-fit of 0.53 with a relative error of 19%. For the asymptomatic group, the 13 indices predicted SAWS with a goodness-of-fit of 0.70 with a relative error of 23%. The GLM models using all 53 geometric indices to predict SAWS did not converge and thus are not presented in the analysis.

## 4 Discussion

Using a novel NN analysis, we attempted to predict aneurysmal wall stress derived from patient-specific computational models, using geometric surrogates. These surrogates (or geometric indices) were unique to either the emergently repaired (symptomatic) or electively repaired (asymptomatic) AAA groups. Using the NN modeling approach, wall stress for the symptomatic and asymptomatic groups exhibited relative errors of 16.8% and 17.4%, respectively (Table 3). Further, the NN-based wall stress prediction exhibited superior model consistency when up to 15 geometric indices were utilized and, comparatively, an improved performance relative to the MARS, GAM, and GLM computational algorithms. These findings underscore the importance and future applications of such high fidelity algorithms for predicting wall stress in AAA computational models.

FEA-based wall stress assessment in AAA models has been frequently reported in literature [6,8,11,13,30,43,44] as a means to assess their rupture risk. However, the use of geometric indices to predict stress is rather limited [5,17,45,46]. Chauhan et al. [14] performed a variable refinement and wall stress prediction similar to the present work. Their work was based on multivariate statistical models to predict SAWS, in contrast with the use of NN models. Further, their study was focused only on emergently repaired AAAs, while the number of variables obtained after correlation analysis was 12, in comparison to the 15 we obtained for the symptomatic AAA group. The prediction accuracy of SAWS in Ref. [14] for the emergently repaired AAAs was similar to the one we report for the symptomatic AAA group (0.76 versus 0.71). Urrutia et al. [21] performed biomechanical and geometric analyses for three groups of AAAs, namely, surveillance (patients not recommended for elective repair), electively repaired, and emergently repaired AAAs. They utilized seven geometric indices for predicting PWS with a goodness-of-fit of 0.58, 0.64, and 0.78, respectively, for the aforementioned groups. In comparison, our number of geometric surrogates is higher, but the prediction accuracy is similar for both studies (0.64 versus 0.71 for electively repaired AAAs; 0.78 versus 0.71 for emergently repaired AAAs). Canchi et al. [47] compared the biomechanical and geometric differences between AAA models generated from Asian and Caucasian patients, and utilized geometric indices to predict 99thWS in a diameter-matched cohort. They found that nine geometric indices for Asian AAAs and seven for Caucasian AAAs were able to predict 99thWS with an accuracy of 0.77 and 0.87, respectively. While we cannot directly compare SAWS with 99thWS, the prediction accuracy reported in Ref. [46] was higher than ours for the asymptomatic AAA group. Our results are rather comparable to those reported by Wu et al. [16], where SAWS derived from electively repaired AAAs can be efficiently predicted by geometric indices with an accuracy of 0.67. In our analysis, the prediction accuracy of SAWS for electively repaired (asymptomatic) AAAs was greater using the NN approach (0.79). One possible explanation for the superiority of NN outcomes over standard statistical methods is the ability to approximate the nonlinear response of the predictors [48]. Additionally, NNs decipher the liaison between the dependent and independent variables, during instances where the inter-relationship between variables is unknown or too complex to handle statistically.

### 4.1 Comparison of Prediction Accuracy for All the Computational Algorithms.

In the symptomatic AAA group, the highest magnitude of the goodness-of-fit was obtained for both NN models—with 53 and 15 indices (R2 of 0.71 for both) (Figs. 7(a) and 7(c)). However, the lowest relative error was exhibited by the NN model with 53 indices. Overall, the lowest and highest relative errors were obtained for the MARS model with 53 indices (13.5%) and the GAM model with 53 indices (239.7%), respectively, in the symptomatic AAA group (Table 3). For the asymptomatic group, the highest goodness-of-fit was exhibited by the NN model with reduced indices (Fig. 7(d)) and the GAM model with reduced indices (Fig. 9(d)) (R2 of 0.79 and 0.80, respectively). Concurrently, the lowest relative error (17.4%) was also reported for the NN model with reduced indices. Conversely, the highest relative error (38.4%) was reported for the GAM model with 53 indices (Table 3).

### 4.2 Neural Network Models Are Superior to Multivariate Adaptive Regression Splines and Generalized Additive Model in Predicting Spatially Averaged Wall Stress.

We compared the prediction abilities of three computational algorithms, namely, NN, MARS, and GAM. We also compared their respective capabilities with the full set of 53 and a reduced set of indices. With the NN approach, the number of input variables or neurons was reduced to 15 in the symptomatic (emergently repaired) AAA group and 13 in the asymptomatic (electively repaired) AAA group (Table 1), which were highly correlated with SAWS. A similar variable refinement procedure in MARS reduced the number of input variables to nine for the symptomatic and eight for the asymptomatic groups, for both MARS and GAM. For the symptomatic group, 53 variables with the MARS algorithm yielded a lower relative error compared to its corresponding reduced set analysis (13.5% versus 17.4%; Table 3). Further, for the asymptomatic group, the variable refinement procedure did not have a significant effect on the MARS-based prediction analysis (22.5% versus 21.7%). Relative errors were reduced up to 5.2 and 1.9 times upon variable refinement in the GAM-based models for the symptomatic and asymptomatic AAA groups, respectively. However, the MARS and GAM predictions did not all have similar goodness-of-fit measures as the NN models (Figs. 79) (R2 = 0.59, 0.70, and 0.71 for the symptomatic group, and R2 = 0.77, 0.80, and 0.79 for the asymptomatic group). This suggests that decreasing the number of input variables has mixed effects on the prediction outcomes of these algorithms. NN exhibited the highest average efficiency in the SAWS prediction for both patient groups; using all 53 variables did not yield a significant compromise in accuracy (Fig. 7). Conversely, using all 53 geometric variables as input for MARS and GAM results in poor stress predictions (Figs. 8 and 9).

### 4.3 Prediction of Spatially Averaged Wall Stress Using Abdominal Aortic Aneurysm Geometry.

In AAA models, maximum wall stress is more strongly associated with curvature measures than maximum aneurysm diameter [5,18,45]. Liljeqvist et al. [46] found that infrarenal aortic volume is strongly associated with AAA growth rate and critical for biomechanical rupture risk assessment. Similarly, Georgakarakos et al. [17] report that AAA mean centerline curvature is a significant predictor of PWS. Geometric measures derived from aneurysm morphology have been shown to be better predictors of rupture risk than the Hardman index [49], a known clinical scoring system utilized prior to endovascular surgeries. Since AAA wall stress is shape dependent, it is reasonable to use geometric indices to predict AAA wall stress in lieu of biomechanical aneurysm models.

Maximum diameter and seven other indices were common to both AAA groups for the reduced set analyses. However, MAA, IPR, THmin, and dc were exclusive to the asymptomatic group. In like manner, Dmaxdir, H, VILT,$γ,$TTmax,and THmode were exclusive to the symptomatic AAA group set. The remaining indices were thickness-based, thrombus-based, and volume-based geometric measures. For both groups, the surface area and the thrombus thickness at the maximum diameter exhibited the corresponding highest positive and negative associations with SAWS, respectively (Table 1). The significance of surface area concurs with the work of Chauhan et al. [14]. In their study, surface area was one of the strong predictors of SAWS for emergently repaired AAAs; however, this measure was not a part of the final, multivariate geometric surrogate model used to predict SAWS. ILT thickness has been directly associated with heightened expression of matrix degrading enzymes that disrupt aneurysmal wall integrity [5052]. Noteworthy is that ILT exhibits a “stress-shield” barrier [53,54], contingent to the degree of thrombus attachment with the arterial wall [52,55]. Biomechanically, an increase in ILT thickness results in lower mean and PWSs in patient-specific AAA FEA models [54]. For a cohort of 100 electively repaired AAA models, the FEA-estimated SAWS was moderately associated with ILT thickness at Dmax ($ρ=−0.383$) [14], which is similar to our results ($ρ=−0.389$). The present work supports the notion that thrombus thickness measures exhibit significant negative associations with SAWS. While Dmax is an important discriminatory index in the correlations, other geometric measures such a wall thickness and thrombus-based indices could be used instead of finite element modeling.

### 4.4 Limitations.

The present study is subject to important limitations, which bound the validity and application of the primary outcome of this research. The validation of the AAA wall stress predicted by geometric measures within the context of clinical outcomes is absent from this work. Therefore, it is possible that clinical symptoms do not necessarily correlate with the computationally predicted wall stresses. FEA modeling requires prior knowledge of meshing and fem software, patient-specific material properties, and the availability of high-performance computers. Thus, the use of AAA wall stress obtained in this manner has limited translational potential. In addition, ILT was not used in the FEA simulations, which would likely decrease the FEA-estimated SAWS, thereby leading to different overall prediction accuracies. Moreover, wall stress was estimated using a generalized neo-Hookean material model, rather than an anisotropic, fiber-based constitutive material model. The strong correlation between SAWS and global geometric indices is attributed to SAWS being a global measure of stress that is averaged over the entire surface of the AAA wall. Hence, SAWS is a global biomechanical parameter and is not location specific. Conversely, PWS and 99thWS are location specific and may be highly correlated with regional (or local) indices such as wall thickness at that location. Further, NN algorithms have been deemed as a “black box” by some authors [58], lacking interpretability of the weights during the model building process [48], and are prone to over fitting [58]. In addition, the assessment of the number of hidden layers, nodes in a network, or the optimal configuration of a NN are not straightforward. NN models perform with high accuracy using large datasets; 50 data points for the symptomatic AAA group and 98 data points for the asymptomatic AAA group are likely not sufficiently large to demonstrate the full potential of such models. Hence, the NN models are expected to have superior performance when used with significantly larger group sizes.

## Acknowledgment

The contributions of Dr. Satish Muluk and Dr. Mark Eskandari overseeing the data collection for this study are gratefully acknowledged.

## Conflict of Interest

The authors have no conflicts of interest to disclose.

## Funding Data

• National Institutes of Health (Grant No. R01HL121293; Funder ID: 10.13039/1000000668).

## Footnotes

1

FFNN refers to the neural network used in image segmentation, which is different from the NN model mentioned throughout the manuscript.

## References

1.
Reimerink
,
J. J.
,
van der Laan
,
M. J.
,
Koelemay
,
M. J.
,
Balm
,
R.
, and
Legemate
,
D. A.
,
2013
, “
Systematic Review and Meta-Analysis of Population-Based Mortality From Ruptured Abdominal Aortic Aneurysm
,”
Br. J. Surg.
,
100
(
11
), pp.
1405
1413
.10.1002/bjs.9235
2.
Martufi
,
G.
,
Lindquist Liljeqvist
,
M.
,
Sakalihasan
,
N.
,
Panuccio
,
G.
,
Hultgren
,
R.
,
Roy
,
J.
, and
Gasser
,
T. C.
,
2016
, “
Local Diameter, Wall Stress, and Thrombus Thickness Influence the Local Growth of Abdominal Aortic Aneurysms
,”
J. Endovasc. Ther.
,
23
(
6
), pp.
957
966
.10.1177/1526602816657086
3.
Hirsch
,
A. T.
,
,
Z. J.
,
Hertzer
,
N. R.
,
Bakal
,
C. W.
,
Creager
,
M. A.
,
Halperin
,
J. L.
,
Hiratzka
,
L. F.
,
Murphy
,
W. R.
,
Olin
,
J. W.
,
Puschett
,
J. B.
,
Rosenfield
,
K. A.
,
Sacks
,
D.
,
Stanley
,
J. C.
,
Taylor
,
L. M.
, Jr.
,
White
,
C. J.
,
White
,
J.
,
White
,
R. A.
,
Antman
,
E. M.
,
Smith
,
S. C.
, Jr.
, and
,
C. D.
, …
Vascular Disease Foundation,
2006
, “
ACC/AHA 2005 Practice Guidelines for the Management of Patients With Peripheral Arterial Disease (Lower Extremity, Renal, Mesenteric, and Abdominal Aortic): A Collaborative Report From the American Association for Vascular Surgery/Society for Vascular Surgery, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, Society of Interventional Radiology, and the ACC/AHA Task Force on Practice Guidelines (Writing Committee to Develop Guidelines for the Management of Patients With Peripheral Arterial Disease): Endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation; National Heart, Lung, and Blood Institute; Society for Vascular Nursing; TransAtlantic Inter-Society Consensus; and Vascular Disease Foundation
,”
Circulation
,
113
(
11
), pp.
463
654
.10.1161/CIRCULATIONAHA.106.174526
4.
Harthun
,
N. L.
,
2008
, “
Current Issues in the Treatment of Women With Abdominal Aortic Aneurysm
,”
Gend. Med.
,
5
(
1
), pp.
36
43
.10.1016/S1550-8579(08)80006-X
5.
Fillinger
,
M. F.
,
Marra
,
S. P.
,
Raghavan
,
M. L.
, and
Kennedy
,
F. E.
,
2003
, “
Prediction of Rupture Risk in Abdominal Aortic Aneurysm During Observation: Wall Stress Versus Diameter
,”
J. Vasc. Surg.
,
37
(
4
), pp.
724
732
.10.1067/mva.2003.213
6.
Vorp
,
D. A.
, and
Vande Geest
,
J. P.
,
2005
, “
Biomechanical Determinants of Abdominal Aortic Aneurysm Rupture
,”
Arterioscler. Thromb. Vasc. Biol.
,
25
(
8
), pp.
1558
1566
.10.1161/01.ATV.0000174129.77391.55
7.
Giannoglou
,
G.
,
Giannakoulas
,
G.
,
Soulis
,
J.
,
Chatzizisis
,
Y.
,
Perdikides
,
T.
,
Melas
,
N.
,
Parcharidis
,
G.
, and
Louridas
,
G.
,
2006
, “
Predicting the Risk of Rupture of Abdominal Aortic Aneurysms by Utilizing Various Geometrical Parameters: Revisiting the Diameter Criterion
,”
Angiology
,
57
(
4
), pp.
487
494
.10.1177/0003319706290741
8.
Vande Geest
,
J. P.
,
Di Martino
,
E. S.
,
Bohra
,
A.
,
Makaroun
,
M. S.
, and
Vorp
,
D. A.
,
2006
, “
A Biomechanics-Based Rupture Potential Index for Abdominal Aortic Aneurysm Risk Assessment: Demonstrative Application
,”
,
1085
(
1
), pp.
11
21
.10.1196/annals.1383.046
9.
Pappu
,
S.
,
Dardik
,
A.
,
Tagare
,
H.
, and
Gusberg
,
R. J.
,
2008
, “
Beyond Fusiform and Saccular: A Novel Quantitative Tortuosity Index May Help Classify Aneurysm Shape and Predict Aneurysm Rupture Potential
,”
Ann. Vasc. Surg.
,
22
(
1
), pp.
88
97
.10.1016/j.avsg.2007.09.004
10.
Martufi
,
G.
,
Di Martino
,
E. S.
,
Amon
,
C. H.
,
Muluk
,
S. C.
, and
Finol
,
E. A.
,
2009
, “
Three-Dimensional Geometrical Characterization of Abdominal Aortic Aneurysms: Image-Based Wall Thickness Distribution
,”
ASME J. Biomech. Eng.
,
131
(
6
), p.
061015
.10.1115/1.3127256
11.
Gasser
,
T. C.
,
Auer
,
M.
,
Labruto
,
F.
,
Swedenborg
,
J.
, and
Roy
,
J.
,
2010
, “
Biomechanical Rupture Risk Assessment of Abdominal Aortic Aneurysms: Model Complexity Versus Predictability of Finite Element Simulations
,”
Eur. J. Vasc. Endovasc. Surg.
,
40
(
2
), pp.
176
185
.10.1016/j.ejvs.2010.04.003
12.
Maier
,
A.
,
Gee
,
M. W.
,
Reeps
,
C.
,
Pongratz
,
J.
,
Eckstein
,
H. H.
, and
Wall
,
W. A.
,
2010
, “
A Comparison of Diameter, Wall Stress, and Rupture Potential Index for Abdominal Aortic Aneurysm Rupture Risk Prediction
,”
Ann. Biomed. Eng.
,
38
(
10
), pp.
3124
3134
.10.1007/s10439-010-0067-6
13.
Polzer
,
S.
, and
Gasser
,
T. C.
,
2015
, “
Biomechanical Rupture Risk Assessment of Abdominal Aortic Aneurysms Based on a Novel Probabilistic Rupture Risk Index
,”
J. R. Soc. Interface
,
12
(
113
), p.
20150852
.10.1098/rsif.2015.0852
14.
Chauhan
,
S. S.
,
Gutierrez
,
C. A.
,
Thirugnanasambandam
,
M.
,
De Oliveira
,
V.
,
Muluk
,
S. C.
,
Eskandari
,
M. K.
, and
Finol
,
E. A.
,
2017
, “
The Association Between Geometry and Wall Stress in Emergently Repaired Abdominal Aortic Aneurysms
,”
Ann. Biomed. Eng.
,
45
(
8
), pp.
1908
1916
.10.1007/s10439-017-1837-1
15.
Leemans
,
E. L.
,
Willems
,
T. P.
,
van der Laan
,
M. J.
,
Slump
,
C. H.
, and
Zeebregts
,
C. J.
,
2017
, “
Biomechanical Markers for Rupture Risk Estimation in Abdominal Aortic Aneurysms
,”
J. Endovasc. Ther.
,
24
(
2
), pp.
254
261
.10.1177/1526602816680088
16.
Wu
,
W.
,
Rengarajan
,
B.
,
Thirugnanasambandam
,
M.
,
Parikh
,
S.
,
Gomez
,
R.
,
De Oliveira
,
V.
,
Muluk
,
S. C.
, and
Finol
,
E. A.
,
2019
, “
Wall Stress and Geometry Measures in Electively Repaired Abdominal Aortic Aneurysms
,”
Ann. Biomed. Eng.
,
47
(
7
), pp.
1611
1625
.10.1007/s10439-019-02261-w
17.
Georgakarakos
,
E.
,
Ioannou
,
C. V.
,
Kamarianakis
,
Y.
,
Papaharilaou
,
Y.
,
Kostas
,
T.
,
Manousaki
,
E.
, and
Katsamouris
,
A. N.
,
2010
, “
The Role of Geometric Parameters in the Prediction of Abdominal Aortic Aneurysm Wall Stress
,”
Eur. J. Vasc. Endovasc. Surg.
,
39
(
1
), pp.
42
48
.10.1016/j.ejvs.2009.09.026
18.
Washington
,
C. B.
,
Shum
,
J.
,
Muluk
,
S. C.
, and
Finol
,
E. A.
,
2011
, “
The Association of Wall Mechanics and Morphology: A Case Study of Abdominal Aortic Aneurysm Growth
,”
ASME J. Biomech. Eng.
,
133
(
10
), p.
104501
.10.1115/1.4005176
19.
Raut
,
S. S.
,
Chandra
,
S.
,
Shum
,
J.
, and
Finol
,
E. A.
,
2013
, “
The Role of Geometric and Biomechanical Factors in Abdominal Aortic Aneurysm Rupture Risk Assessment
,”
Ann. Biomed. Eng.
,
41
(
7
), pp.
1459
1477
.10.1007/s10439-013-0786-6
20.
Raut
,
S. S.
,
Chandra
,
S.
,
Shum
,
J.
,
Washington
,
C. B.
,
Muluk
,
S. C.
,
Finol
,
E. A.
, and
Rodriguez
,
J. F.
,
2013
, “
Biological, Geometric and Biomechanical Factors Influencing Abdominal Aortic Aneurysm Rupture Risk: A Comprehensive Review
,”
Recent Pat. Med. Imaging
,
3
(
1
), pp.
44
59
.10.2174/1877613211303010006
21.
Urrutia
,
J.
,
Roy
,
A.
,
Raut
,
S. S.
,
Anton
,
R.
,
Muluk
,
S. C.
, and
Finol
,
E. A.
,
2018
, “
Geometric Surrogates of Abdominal Aortic Aneurysm Wall Mechanics
,”
Med. Eng. Phys.
,
59
, pp.
43
49
.10.1016/j.medengphy.2018.06.007
22.
Milos
,
R.
,
Dejan
,
P.
, and
,
F.
,
2011
, “
Mining Data From CFD Simulation for Aneurysm and Carotid Bifurcation Models
,”
Conference Proceedings—IEEE Engineering in Medicine and Biology Society
, Boston, MA, Aug. 30–Sept. 3, pp.
8311
8314
.10.1109/IEMBS.2011.6092049
23.
Blagojević
,
M. D.
,
,
M. D.
,
,
M. M.
, and
Filipović
,
N. D.
,
2015
, “
Neural Network Based Approach for Predicting Maximal Wall Shear Stress in the Artery
,” 2015 IEEE 15th International Conference on Bioinformatics and Bioengineering (
BIBE
), Belgrade, Serbia, Nov. 2–4, pp.
1
5
.10.1109/BIBE.2015.7367713
24.
Soudah
,
E.
,
Rodriguez
,
J. F.
, and
Lopez
,
R.
,
2015
, “
Mechanical Stress in Abdominal Aortic Aneurysms Using Artificial Neural Networks
,”
J. Mech. Med. Biol.
,
15
(
03
), p.
1550029
.10.1142/S0219519415500293
25.
,
A.
,
Bakhaty
,
A.
,
Kim
,
J.
,
Mubarak
,
Y.
, and
,
M.
,
2019
, “
Bridging Finite Element and Machine Learning Modeling: Stress Prediction of Arterial Walls in Atherosclerosis
,”
ASME J. Biomech. Eng.
,
141
(
8
), p.
084502
.10.1115/1.4043290
26.
Shum
,
J.
,
2011
,
Risk Assessment of Abdominal Aortic Aneurysms by Geometry Quantification Measures
,
Carnegie Mellon University
,
Pittsburgh, PA
.
27.
Raut
,
S. S.
,
Jana
,
A.
,
De Oliveira
,
V.
,
Muluk
,
S. C.
, and
Finol
,
E. A.
,
2014
, “
The Effect of Uncertainty in Vascular Wall Material Properties on Abdominal Aortic Aneurysm Wall Mechanics
,”
Computational Biomechanics for Medicine
,
B.
Doyle
,
K.
Miller
,
A.
Wittek
, and
P. M. F.
Nielsen
, eds.,
Springer New York
,
New York
, pp.
69
86
.
28.
Raut
,
S. S.
,
2012
,
Patient-Specific 3D Vascular Reconstruction and Computational Assessment of Biomechanics—An Application to Abdominal Aortic Aneurysm
,
Carnegie Mellon University
,
Pittsburgh, PA
.
29.
Raut
,
S. S.
,
Liu
,
P.
, and
Finol
,
E. A.
,
2015
, “
An Approach for Patient-Specific Multi-Domain Vascular Mesh Generation Featuring Spatially Varying Wall Thickness Modeling
,”
J. Biomech.
,
48
(
10
), pp.
1972
1981
.10.1016/j.jbiomech.2015.04.006
30.
Venkatasubramaniam
,
A. K.
,
Fagan
,
M. J.
,
Mehta
,
T.
,
Mylankal
,
K. J.
,
Ray
,
B.
,
Kuhan
,
G.
,
Chetter
,
I. C.
, and
McCollum
,
P. T.
,
2004
, “
A Comparative Study of Aortic Wall Stress Using Finite Element Analysis for Ruptured and Non-Ruptured Abdominal Aortic Aneurysms
,”
Eur. J. Vasc. Endovasc. Surg.
,
28
(
2
), pp.
168
176
.10.1016/j.ejvs.2004.03.029
31.
Raghavan
,
M. L.
, and
Vorp
,
D. A.
,
2000
, “
Toward a Biomechanical Tool to Evaluate Rupture Potential of Abdominal Aortic Aneurysm: Identification of a Finite Strain Constitutive Model and Evaluation of Its Applicability
,”
J. Biomech.
,
33
(
4
), pp.
475
482
.10.1016/S0021-9290(99)00201-8
32.
Raghavan
,
M. L.
,
Kratzberg
,
J.
,
Castro de Tolosa
,
E. M.
,
Hanaoka
,
M. M.
,
Walker
,
P.
, and
da Silva
,
E. S.
,
2006
, “
Regional Distribution of Wall Thickness and Failure Properties of Human Abdominal Aortic Aneurysm
,”
J. Biomech.
,
39
(
16
), pp.
3010
3016
.10.1016/j.jbiomech.2005.10.021
33.
Shum
,
J.
,
DiMartino
,
E. S.
,
Goldhamme
,
A.
,
Goldman
,
D. H.
,
Acker
,
L. C.
,
Patel
,
G.
,
Ng
,
J. H.
,
Martufi
,
G.
, and
Finol
,
E. A.
,
2010
, “
Semiautomatic Vessel Wall Detection and Quantification of Wall Thickness in Computed Tomography Images of Human Abdominal Aortic Aneurysms
,”
Med. Phys.
,
37
(
2
), pp.
638
648
.10.1118/1.3284976
34.
Shum
,
J.
,
Xu
,
A.
,
Chatnuntawech
,
I.
, and
Finol
,
E. A.
,
2011
, “
A Framework for the Automatic Generation of Surface Topologies for Abdominal Aortic Aneurysm Models
,”
Ann. Biomed. Eng.
,
39
(
1
), pp.
249
259
.10.1007/s10439-010-0165-5
35.
Lee
,
K.
,
Zhu
,
J.
,
Shum
,
J.
,
Zhang
,
Y.
,
Muluk
,
S. C.
,
Chandra
,
A.
,
Eskandari
,
M. K.
, and
Finol
,
E. A.
,
2013
, “
Surface Curvature as a Classifier of Abdominal Aortic Aneurysms: A Comparative Analysis
,”
Ann. Biomed. Eng.
,
41
(
3
), pp.
562
576
.10.1007/s10439-012-0691-4
36.
Wood
,
S.
,
2017
,
Generalized Additive Models: An Introduction With R
, 2nd ed.,
Chapman and Hall/CRC
, Boca Raton, FL.
37.
Milborrow
,
S.
,
2011
, “
Derived From MDA: MARS by T. Hastie and R. Tibshirani
,”
,
R Package
.https://cran.rproject.org/web/packages/earth/earth.pdf
38.
Chen
,
S. Y.
,
Feng
,
Z.
, and
Yi
,
X.
,
2017
, “
A General Introduction to Adjustment for Multiple Comparisons
,”
J. Thorac. Dis.
,
9
(
6
), pp.
1725
1729
.10.21037/jtd.2017.05.34
39.
R Core Team
,
2018
, “
R: A Language and Environment for Statistical Computing
,”
R Foundation for Statistical Computing
, Vienna, Austria, accessed Jan. 7, 2021, www.R-project.org/
40.
Wright
,
K.
,
2018
, “
Corrgram: Plot a Correlogram
,” accessed Jan. 8, 2021, https://cran.r-project.org/web/packages/corrgram/index.html
41.
Hastie
,
T.
, and
Hastie
,
M. T.
,
2018
, Package ‘gam’. GAM Package CRAN, accessed Jan. 15, 2020, https://cran.r-project.org/web/packages/gam/gam.pdf
42.
R Core Team,
2013
,
R: A Language and Environment for Statistical Computing
,
R Foundation for Statistical Computing
, Vienna, Austria.
43.
Raghavan
,
M. L.
,
Fillinger
,
M. F.
,
Marra
,
S. P.
,
Naegelein
,
B. P.
, and
Kennedy
,
F. E.
,
2005
, “
Automated Methodology for Determination of Stress Distribution in Human Abdominal Aortic Aneurysm
,”
ASME J. Biomech. Eng.
,
127
(
5
), pp.
868
871
.10.1115/1.1992530
44.
Polzer
,
S.
,
Gasser
,
T. C.
,
Vlachovsky
,
R.
,
Kubicek
,
L.
,
Lambert
,
L.
,
Man
,
V.
,
Novak
,
K.
,
Slazansky
,
M.
,
Bursa
,
J.
, and
Staffa
,
R.
,
2020
, “
Biomechanical Markers Are More Sensitive Than Diameter in Predicting Rupture of Asymptomatic Abdominal Aortic Aneurysms
,”
J. Vasc. Surg.
,
71
(
2
), pp.
617
626
.10.1016/j.jvs.2019.03.051
45.
Elger
,
D. F.
,
Blackketter
,
D. M.
,
Budwig
,
R. S.
, and
Johansen
,
K. H.
,
1996
, “
The Influence of Shape on the Stresses in Model Abdominal Aortic Aneurysms
,”
ASME J. Biomech. Eng.
,
118
(
3
), pp.
326
332
.10.1115/1.2796014
46.
Liljeqvist
,
M. L.
,
Hultgren
,
R.
,
Gasser
,
T. C.
, and
Roy
,
J.
,
2016
, “
Volume Growth of Abdominal Aortic Aneurysms Correlates With Baseline Volume and Increasing Finite Element Analysis-Derived Rupture Risk
,”
J. Vasc. Surg.
,
63
(
6
), pp.
1434
1442
.10.1016/j.jvs.2015.11.051
47.
Canchi
,
T.
,
Patnaik
,
S. S.
,
Nguyen
,
H. N.
,
Ng
,
E. K. Y.
,
Narayanan
,
S.
,
Muluk
,
S. C.
,
De Oliveira
,
V.
, and
Finol
,
E. A.
,
2020
, “
A Comparative Study of Biomechanical and Geometrical Attributes of Abdominal Aortic Aneurysms in the Asian and Caucasian Populations
,”
ASME J. Biomech. Eng.
,
142
(
6
), p. 061003.10.1115/1.4045268
48.
Paliwal
,
M.
, and
Kumar
,
U. A.
,
2009
, “
Neural Networks and Statistical Techniques: A Review of Applications
,”
Expert Syst. Appl.
,
36
(
1
), pp.
2
17
.10.1016/j.eswa.2007.10.005
49.
Hollingsworth
,
A. C.
,
Dawkins
,
C.
,
Wong
,
P. F.
,
Walker
,
P.
,
Milburn
,
S.
, and
Mofidi
,
R.
,
2019
, “
Aneurysm Morphology is a More Significant Predictor of Survival Than Hardman's Index in Patients With Ruptured or Acutely Symptomatic Abdominal Aortic Aneurysms
,”
Ann. Vasc. Surg.
,
58
, pp.
222
231
.10.1016/j.avsg.2018.10.018
50.
Khan
,
J. A.
,
Abdul Rahman
,
M. N.
,
Mazari
,
F. A.
,
Shahin
,
Y.
,
Smith
,
G.
,
,
L.
,
Fagan
,
M. J.
,
Greenman
,
J.
,
McCollum
,
P. T.
, and
Chetter
,
I. C.
,
2012
, “
Intraluminal Thrombus Has a Selective Influence on Matrix Metalloproteinases and Their Inhibitors (Tissue Inhibitors of Matrix Metalloproteinases) in the Wall of Abdominal Aortic Aneurysms
,”
Ann. Vasc. Surg.
,
26
(
3
), pp.
322
329
.10.1016/j.avsg.2011.08.015
51.
Koole
,
D.
,
Zandvoort
,
H. J.
,
Schoneveld
,
A.
,
Vink
,
A.
,
Vos
,
J. A.
,
van den Hoogen
,
L. L.
,
de Vries
,
J. P.
,
Pasterkamp
,
G.
,
Moll
,
F. L.
, and
van Herwaarden
,
J. A.
,
2013
, “
Intraluminal Abdominal Aortic Aneurysm Thrombus is Associated With Disruption of Wall Integrity
,”
J. Vasc. Surg.
,
57
(
1
), pp.
77
83
.10.1016/j.jvs.2012.07.003
52.
Piechota-Polanczyk
,
A.
,
Jozkowicz
,
A.
,
Nowak
,
W.
,
Eilenberg
,
W.
,
Neumayer
,
C.
,
Malinski
,
T.
,
Huk
,
I.
, and
Brostjan
,
C.
,
2015
, “
The Abdominal Aortic Aneurysm and Intraluminal Thrombus: Current Concepts of Development and Treatment
,”
Front. Cardiovasc. Med.
,
2
, p.
19
.10.3389/fcvm.2015.00019
53.
Thubrikar
,
M. J.
,
Robicsek
,
F.
,
Labrosse
,
M.
,
Chervenkoff
,
V.
, and
Fowler
,
B. L.
,
2003
, “
Effect of Thrombus on Abdominal Aortic Aneurysm Wall Dilation and Stress
,”
J. Cardiovasc. Surg. (Torino)
,
44
(
1
), pp.
67
77
.https://www.minervamedica.it/en/journals/cardiovascular-surgery/article.php?cod=R37Y2003N01A0067
54.
Haller
,
S. J.
,
Crawford
,
J. D.
,
Courchaine
,
K. M.
,
Bohannan
,
C. J.
,
Landry
,
G. J.
,
Moneta
,
G. L.
,
Azarbal
,
A. F.
, and
Rugonyi
,
S.
,
2018
, “
Intraluminal Thrombus is Associated With Early Rupture of Abdominal Aortic Aneurysm
,”
J. Vasc. Surg.
,
67
(
4
), pp.
1051
1058.e1051
.10.1016/j.jvs.2017.08.069
55.
Meyer
,
C. A.
,
Guivier-Curien
,
C.
, and
Moore
, and
J. E.
, Jr.
,
2010
, “
Trans-Thrombus Blood Pressure Effects in Abdominal Aortic Aneurysms
,”
ASME J. Biomech. Eng.
,
132
(
7
), p.
071005
.10.1115/1.4001253
56.
Erhart
,
P.
,
Lindquist Liljeqvist
,
M.
,
Roy
,
J.
,
Gasser
,
T. C.
, and
Böckler
,
D.
,
2019
, “
Does Finite Element Analysis Enable us to Predict Rupture? The Benefits and Limitations of Finite Element Analysis in Patients With Abdominal Aortic Aneurysms
,”
Endovasc. Today
, 18(3), pp.
74
78
.https://evtoday.com/pdfs/et0319_F5_Erhart.pdf
57.
Miller
,
K.
,
Mufty
,
H.
,
Catlin
,
A.
,
Rogers
,
C.
,
Saunders
,
B.
,
Sciarrone
,
R.
,
Fourneau
,
I.
,
Meuris
,
B.
,
Tavner
,
A.
, and
Joldes
,
G. R.
,
2019
, “
Wall Stress Distributions in Abdominal Aortic Aneurysms Do Not Correlate With Symptoms
,” arXiv Preprint arXiv:1904.07393.https://arxiv.org/ftp/arxiv/papers/1904/1904.07393.pdf
58.
Tu
,
J. V.
,
1996
, “
Advantages and Disadvantages of Using Artificial Neural Networks Versus Logistic Regression for Predicting Medical Outcomes
,”
J. Clin. Epidemiol.
,
49
(
11
), pp.
1225
1231
.10.1016/S0895-4356(96)00002-9


text:
TEXT SIZE

CrossRef (0)
Interval prediction on the sum of binary random variables indexed by a graph

Seongoh Parka, Kyu S. Hahnb, Johan Lima, and Won Son1,c

aDepartment of Statistics, Seoul National University, Korea, bDepartment of Communication, Seoul National University, Korea, cThe Bank of Korea, Korea
Correspondence to: 1Economist, The Bank of Korea, 67, Sejong-daero, Jung-gu, Seoul 04514, Korea. E-mail: son.won@gmail.com
Received October 15, 2018; Revised March 7, 2019; Accepted March 28, 2019.
Abstract

In this paper, we propose a procedure to build a prediction interval of the sum of dependent binary random variables over a graph to account for the dependence among binary variables. Our main interest is to find a prediction interval of the weighted sum of dependent binary random variables indexed by a graph. This problem is motivated by the prediction problem of various elections including Korean National Assembly and US presidential election. Traditional and popular approaches to construct the prediction interval of the seats won by major parties are normal approximation by the CLT and Monte Carlo method by generating many independent Bernoulli random variables assuming that those binary random variables are independent and the success probabilities are known constants. However, in practice, the survey results (also the exit polls) on the election are random and hardly independent to each other. They are more often spatially correlated random variables. To take this into account, we suggest a spatial auto-regressive (AR) model for the surveyed success probabilities, and propose a residual based bootstrap procedure to construct the prediction interval of the sum of the binary outcomes. Finally, we apply the procedure to building the prediction intervals of the number of legislative seats won by each party from the exit poll data in the 19th and 20th Korea National Assembly elections.

Keywords : binary sum, exit poll, graph indexed variables, Korea National Assembly election, prediction interval, residual bootstrap, spatial auto-regressive model
1. Introduction

In this paper, we are interested in the interval prediction of the sum of dependent binary random variables indexed by a graph. Suppose an undirected graph G = (V, E) is given, where V is the set of vertices and E is the set of edges e = (v,w) with v,wV, and observe data ${(Pv,Xv),v∈V}$ on the graph. The observation pv of Pv is the success probability for the binary outcome on the vertex v and Xv is an exogenous covariate that influences Pv. The statistic we are interested in is the weighted sum of binary random variables Yvs,

$T=∑v∈VwvYv,$

where wv is a pre-decided weight for the vertex v and, given ${(Pv,Xv),v∈V},Yv$ are independently from a Bernoulli distribution with success probabilities $pvs$. We construct 100(1 − α)% prediction interval of T when we have observations ${(Pv,Xv),v∈V}$.

The problem above often arises in prediction problem in various elections in many countries. One example is the United States Electoral College for the US presidential election, where v is the index of the state which forms the graph along with its spatial location; wv is the number of Electoral College (EC) vote of the state v; Pv is the winning probability of a candidate of interest at state v that is available from the survey or exit poll; and Xv is the extra covariate that can influence on Pv. We are interested in predicting the number of EC votes won by the candidate using the survey results. Another example, from which this paper is actually motivated, is the election for the Korean National Assembly (KNA), where v is the election district that forms a graph with its spatial location; wv is set as 1 (one seat for one district); and T is the number of congress seats won by a party of interest in the election. The success probabilities $Pvs$ are evaluated from the exit poll, and $Xvs$ are chosen to explain the regionalism (whose existence in Korean politics is known for many decades). Here, we are interested in making interval prediction for the number of congressional seats taken by a party.

A traditional and popular method to construct the prediction interval of T is by assuming that $Pvs$ are known constants and generating many independent (over the vertex set V) Bernoulli random variables with success probabilities $Pvs$. This method has been the practice for the exit polls for KNA election since 2008’s election. However, the data example in Section 4 shows that the independent Bernoulli method disregards both the randomness of $Pvs$ and the spatial correlations among them. In sequel, it underestimates the variance of T and results in a shorter prediction interval. The variance of T is decomposed as

$var(T)=var{E(T|P)}+E{var(T|P)}=var(∑v∈VPv)+E{∑v∈VPv(1-Pv)},$

where the existing methods disregard the first term and approximate the second to $∑v∈VPv(1-Pv)$ with the surveyed supporting rate $pvs$. Thus, the existing methods underestimate the variance of T, if $Pvs$ are random.

This is one of many reasons for the failure of the exit poll for the KNA election. We remark that the exit poll for the KNA election starts in year 2004 (the 17th election) and the prediction interval for the number of the congressional seats is first given in year 2008 (the 18th election).

In this paper, we propose a new method to build a prediction interval of the sum statistic T. Our new way is a resampling based procedure. It assumes the spatial auto-regressive (AR) model for $Pvs$ (more precisely for $Zv=log{Pv/(1-Pv)}$) and adapts the residual bootstrap to get re-samples of T. If Pv = 0 or 1, the logarithm is not well-defined. We may use the perturbed logit function defined by $log(a/(1-a))ifPv=0$ and $log(a/(1-a)/a)ifPv=1$ where a is a very small constant such as 1e − 10. The prediction interval is directly from the resamples.

The new method is applied to the exit polls of the 19th and 20th KNA elections and compared with the independent Bernoulli method. The new method provides a wider interval than the independent Bernoulli method as expected, but contains the true number of seats (observed as the outcomes of the election) within the interval in both the 19th and 20th elections.

The remainder of the paper is organized as follows. In Section 2, we introduce the spatial AR model assumed for the observations ${(Pv,Xv),v∈V}$ and the iterative procedure to estimate the model parameters. The bootstrap procedure to build the prediction interval for T is proposed in Section 3. In Section 4, we start the section with a brief introduction to the exit poll and the history of the exit poll for the KNA election. We then apply our method to the exit polls of the 19th and 20th KNA elections, and compare the results to those of the independent Bernoulli method. In Section 5, we conclude the paper with a brief summary of the work.

2. Spatial model and estimation

In this section, we introduce the spatial AR model and the estimation procedure of the model parameters. Recall that $G=(V,E)$ is the graph on which ${ (Pv,Xv),∈V }$ are defined. Suppose we let $Zv=log{Pv/(1-Pv)}$, and $N(v)={w∈V|(v,w)∈E}$ be the neighborhood of the vertex v on the graph. For Zv, we assume the spatial AR model, that is,

$Zv=β0+XvTβ+Uv,Uv=∑w∈N(v)ρv,wUw+ϵv,$

where $β=(β1,β2,...,βp)T∈Rp$ (assume the dimension of the exogenous covariate vector is p) is the regression coefficient vector, $ρv.w ∈ R$ is the spatial AR coefficient, and $ϵvs$ are independent measurement errors having mean 0 and variance $σϵ2$. The model above is one of popular models popular models in spatial regression (Anselin, 1988; Baltagi et al., 2003; Arnold and Wied, 2010). The model has too many parameters and is rarely estimable from the observation. In practice, we make further structural assumption on the spatial AR coefficients ${ρv.w,v,w∈V}$. Some examples are (i) coefficients are constant as $ρv.w=ρ$ for all $v,w∈V$, (ii) $ρv.w=ρ/dv$ with $dv=|{w∈V,(v,w)∈E}|$, or (iii) $ρv.w=ρ1I{w∈N1(v)}+ρ2I{w∈N2(v)}$ with different neighborhoods N1(v), N2(v) of v. In below, for notational simplicity, we assume ρv.w = ρ for all v, wV, that is, the case (i), together with an assumption that I−A(ρ) is positive-definite. The condition in terms of the matrix is typically made in many literatures (Anselin, 1988; Lee, 2002), which naturally restricts the feasible range of ρ. It is required to guarantee a well-defined covariance matrix and to define Zv well from Uv in (2.1), or vice versa.

When the observations {(pv, Xv), vV} (equivalently, {(zv, Xv), vV}) are given, a computationally attractive procedure to estimate the model parameters is the iterative least square method (Hordijk, 1974; Cochrane and Orcutt, 1949; Ord, 1975), which iteratively updates the estimate of (β0, βT)T and (ρ, $σϵ2$). In the first step, given the estimate of ρ and $σϵ2$, the variance-covariance matrix of U = $(U1,...,U|V|)T$ in (2.1) is

$var(U)=(I-A(ρ))-2σϵ2,$

where A(ρ) is a $|V|$×$|V|$ symmetric adjacency matrix having ρ at the (u, v)th element if (u, v) ∈ E, and 0, otherwise. The estimate of the regression coefficient can then be obtained by solving the generalized least squares problem as

$(β^0,β^T)T={XTvar(U)-1X}-1 XTvar(U)-1z,$

where X is a $|V|$ × (p + 1) matrix whose vth row is (1, $XvT$) and z = $(z1,z2,...,z|V|)T$. In the second step, to update ρ, we do the multiple regression, where the response is the residual from the first step $U˜v=zv-β^0-XvTβ^$ and the covariate $∑w∈N(v)U˜v$ for vV. Here, the covariate vector can be changed depending on the structural assumption on spatial AR coefficients $ρv.ws$. We iterate the above two steps until the parameter estimates converge.

3. Prediction interval with spatial bootstrap

In this section, we introduce the new method to build the 100(1 − α)% prediction interval for the sum statistic using the spatial bootstrap. For simplicity of notation, we take off hats from parameter estimates unless it is confusing. Our procedure begins by computing residuals

$ϵv=zv-β0-XvTβ-ρ∑w∈N(v)(zw-β0-XwTβ), v∈V.$

We let $ϵv*$ be a bootstrapped residual which is sampled uniformly from {ϵw,wV} for every vV. Then, a resampled copy can be obtained

$z*=X(β0β)+(I-A(ρ))-1ϵ*,$

where $ϵ*=(ϵ1*,…,ϵ|V|*)T$, which is a vector representation of equation (2.1). Now, given success probabilities {$pv*$, vV}, the sum of Bernoulli variables can be evaluated empirically, which is easy to implement using existing software. Exploiting the Monte Carlo method, we obtain |V| many Bernoulli trials {$Yv*$, vV}, the sum of which is $T*=∑v∈VYv*$. We repeat this procedure as many times as required, say B times. To make it clear, it is summarized in Algorithm 1.

We propose a method to build the prediction interval of T using the empirical $100(α/2)th$ and $100(1-α/2)th$ percentiles of the bootstrapped sums $T(b)=∑v∈VYv(b)$, b = 1, 2, . . . , B, denoted by $Tα/2$ and $T1-α/2$. Thus, our proposal is

$[Tα2,T1-α2].$

Our proposal can be understood as a procedure with a t-type statistic that is

$tind=T-μ^(T)se.ind^(T)=T-∑v∈Vpv∑v∈Vpv(1-pv),$

where T is the number of seats we want to predict, $μ^(T)=∑v∈Vpv$ is the estimated expected number of seats from the observations {pv, vV}, and $se.ind^(T)$ is the estimated standard deviation of T under the misspecified independent assumption among $Pvs$. The distribution of the above tind is unknown even either under normality assumption on $Pvs$ or in asymptotic due to the correlation of $Pvs$ over the graph G = (V, E). We propose the spatial bootstrap samples {$pv(b)$, vV} to approximate the distribution tind. Using {T(b), b = 1, 2, . . . , B} obtained as described in Algorithm 1, we evaluate the statistics $tind(b)=T(b)-μ^(T)se.ind^(T), b=1,2,…,B,$ where $μ^(T)$ and $se.ind^(T)$ are those defined in (3.4). Suppose tu and t are the upper, lower $100(α/2)th$ empirical quantiles of {$tind(b)$, b = 1, 2, . . . , B}. Then, our proposal (3.3) is equivalent to

$[μ^(T)+tℓse.ind^(T),μ^(T)+tuse.ind^(T)]=[∑v∈Vpv+tℓ∑v∈Vpv(1-pv),∑v∈Vpv+tu∑v∈Vpv(1-pv)].$
4. Application to exit polls of KNA election

In this section, we apply the method proposed in Section 2 and 3 to building the prediction interval of the number of seats of each party from the exit poll data in two most recent Korean National Assembly elections, the 19th and 20th elections in year 2012 and 2016. However, we only show the results of the 19th election, in which the exit poll fails to predict the true number of seats for several reasons (Kwak et al., 2013), and defer the results of the 20th election to the Appendix.

### 4.1. The exit polls and the KNA election

We start with a brief introduction of election exit polls that have been widely used in the U.S. since the 1970s. Their use has now expanded to other democracies (Mitosfky, 1991, 1995; Greiner and Quinn, 2010;Wang et al., 2015). Exit polls are interviews conducted with voters as they leave polling booths. The expansion of exit polls has been driven by broadcasters’ competition to be the first to declare election results. In the exit poll, predicting the outcomes of presidential elections is fairly straightforward because the popular vote count is of primary concern. However, popular votes are of secondary concern in legislative elections. Instead, what is of key interest is the number of seats each party wins whereas the outcomes of individual races are of often secondary importance (Brown and Payne, 1975; Curtice and Firth, 2008; Bafumi et al., 2010). Therefore, news media’s post-election announcement of exit poll results focus on the number of seats predicted to be won by each party. Given this objective, the news media therefore aims to find an approximate prediction interval of the number of seats, not only its point estimate.

The exit poll data we analyze in this section is obtained during the 19th KNA election held on April, 17, 2012. The KNA is a 300-member unicameral legislature. The KNA election has been held every four years since the promulgation of the March 1988 Electoral Law. The KNA is a Supplementary Member system, where 246 members are elected from constituencies while 54 members are elected at the national level through proportional representation (PR). In the 2012 KNA election, three broadcasting networks (KBS, MBC, and SBS) jointly conducted exit polls. They contract three major polling firms and hire approximately 13,000 interviewers and 500 overseers. Total expenditure by the three networks exceeded seven million U.S. dollars.

The pollsters employ a two-stage cluster design (Mendenhall et al., 1971). In the first stage, random samples of 2,484 polling stations are selected across the nation. In the second stage, one in every five voters is sampled for a face-to-face interview at each polling station. Interviews are conducted between 6am. and 5pm., whereas the polling booths close at 6pm. The exit polls end early because the networks need time to tabulate the votes so that the results can be announced immediately upon the closure of polling stations. Aside from respondents’ voting choice, their age and gender are recorded. In the 19th election, this procedure yielded 674,819 completed interviews. The overall non-response rate was 17.4%. In their post-election announcement, three networks report that the station-specific margin of error ranged between ±2.2% and ±5.1% across 2,484 individual polling stations.

### 4.2. The spatial graph structure and model

In our exit poll example, the 246 election districts consist of the vertex set V and their spatial locations define the edge set. If two election districts u, vV are spatially neighboring each other, we define e(u, v) = 1 and 0 otherwise. About the neighboring system, we further divide it into two types depending on the administrative district (the case (iii) mentioned in Section 2). Here, we consider the metropolitan/province level administrative district. There are 17 metropolitan/province level administrative districts, each of which contains from one to dozens of election districts; each election district is for 1 number of legislative seat. The two types of neighboring system are written as follows. First, let

$N(v)={w∈V,e(v,w)=e(w,v)=1}$

be the set of all districts neighboring with v. We define the within and between neighborhoods, denoted as Nwtn(v) and Nbtw(v), by

$Nwtn(v)={w∈N(v),w,v are in the same administrative district},Nbtw(v)={w∈N(v),w,v are not in the same administrative district}.$

According to the above two types of neighboring system, we define the spatial model (2.1) for the exit poll data. We have the wining probability of a given party for each district v evaluated from the exit poll data, say pv, vV. We define $Zv=log{Pv/(1-Pv)}$ and assume the model (2.1) with the graph G = (V, E) defined above. In the model (2.1), Xv denotes 16 dummy variables indicating 17 administrative districts, where Seoul district is set as the reference district. The spatial coefficient ρv.w is defined as

$ρv,w={ρwtn,if w∈Nwtn(v),ρbtw,if w∈Nbtw(v),0,otherwise.$

Thus, the model for spatial latent variables in (2.1) becomes

$Uv=ρwtn∑w∈Nwtn(v)Uw+ρbtw∑w∈Nbtw(v)Uw+ϵv.$

Let Awtn be the adjacency matrix in $R|V|×|V|$ for the within neighborhood so that it has $ρwtn$ at (v,w) if vNwtn(w) or wNwtn(v), and 0 otherwise, and Abtw defined in the same manner. A pair of spatial correlations ($ρwtn$, $ρbtw$) should range in the set where I − AwtnAbtw is positive definite.

### 4.3. Existing methods for prediction interval

Two existing methods that are known to be used in practice are the normal approximation (NoA) and the Monte Carlo (MCind) approximation by Huh (2008) under the assumption of independence among the observed pvs.

It is followed by the classical central limit theorem that the sum of n independent (heterogeneous) Bernoulli variables behave like the Gaussian random variable when n is sufficiently large and some regularity conditions are assumed. Consequently, based on the asymptotic normality of the pivotal statistic (3.4), the confidence interval for T of level 1 − α is given by

$[μ^(T)-zα2 se.ind^(T),μ^(T)+zα2 se.ind^(T)],$

where zγ (0 < γ < 1) is the 100(1 − γ)th quantile of the standard normal distribution.

On the other hand, MCind approximation is based on independent Bernoulli random samples {$Yv(b).ind$, vV} with observed success probabilities {pv, vV}. The suggested prediction interval of level 1 − α has its endpoints by $100(α/2)$, $100(1-α/2)th$ quantiles of $T(b).ind=∑v∈VYv(b).ind$, b = 1, 2, . . . , B. However, unlike ours, this does not consider the variability and spatial coherency in {pv : vV}, and its length tends to be shorter than ours.

### 4.4. Results: the 19th election

In the 19th election, two major parties, Saenuri Party (SNP) and Minju Tonghap Party (MTP), competed with each other (there are a few more parties, but their numbers of seats are too small to be included); subsequently, the two parties won 152 and 127 number of seats, respectively. Table 1 reports the prediction using the exit poll data by three broadcasting systems. The table also shows that KBS predicted SNP and MTP would win seats between (131, 147) and (131, 147), respectively; MBC announced that the two parties would win seats between (126, 151) and (128, 150), respectively; finally, SBS predicted that the two parties would win seats between (130, 153) and (128, 153), respectively. Here, the prediction intervals are officially based on independent Bernoulli trials. The three broadcasting networks use the same data from the exit poll after the same adjustment procedure for the non-response. However, three networks do additional adjustment to the results and their final results are different. Their adjustments for the non-response are not precise and introduce a significant bias in both point and interval predictions. In the analysis below, we use the adjustment procedure which the authors develop for the MBC system during the 20th KNA election.

We apply the proposed spatial bootstrap method (SB) to build the prediction intervals as well as compare the results to the NoA and MCind. In the analysis below, three methods are applied to each political party, SNP and MTP, independently, to evaluate the expected number of seats and its prediction interval for each party. The size of the Monte Carlo samples for the MCind and SB is set at 10,000.

Table 2 reports the predictions on the number of seats by three methods. We find that the prediction interval by the SB method wider than the two existing methods. The SB accounts for both the spatial dependence and the effects of exogenous administrative districts which makes the interval wider than those based on the independent assumption without considering the administrative districts’ effect; consequently, the true numbers of seats fall within the intervals by the SB.

The covariate effects, the effects of administrative districts, are plotted in Figure 1, where the coefficient (effect) of administrative district “Seoul” is fixed as 0 for the comparison. The figure shows that the SNP has positive effect in the east and south east part of Korea, whereas the MTP does in the south west part of Korea.

The estimated spatial coefficients are reported in Table 3, where the standard errors are estimated from the bootstrap replications and the p-values are based on the normal distribution. The results show no significant within-spatial-correlation once we account for the covariate effect of the administrative districts. In particular, for the MTP, the covariate effects of the administrative districts are different among districts (Figure 1). The two neighboring election districts from different administrative districts are marginally correlated, but look negatively correlated once we adjust the effects of the administrative districts. We conjecture this would be the reason of the negative estimate of $ρ^btw$ of the MTP.

### 4.5. Small area prediction

In the election, it is often a particular interest to see a specific region with small number of electoral precincts such as the Nakdonggang River belt that refers to 8 districts around the Nakdonggang River in western Busan. The seat prediction in the small area WV is straightforward since the results can be derived as a byproduct of Algorithm 1. Replacing V by its subset W in the algorithm provides the desired result. Table 4 shows the predictive interval for the Nakdonggang River belt and Seoul city with 8 and 48 legislative seats.

5. Summary

The primary goal of exit polls and legislative elections is to predict the number of seats won by major parties. However, they are not very accurate despite the large amount of financial resources dispatched to conducting exit polls. Furthermore, no formal procedures are suggested to build the prediction intervals of the number of seats won by each party. In this work, we recast the problem into a more general problem: the prediction of the sum of binary random variables on the graph when their success probabilities are observable. We consider the AR regression model to account for the effect of exogenous covariates on the graph and the spatial dependence over the graph. We propose a spatial bootstrap procedure to build the prediction interval of the sum along with the AR regression model. We apply our procedure to the exit poll data from the 19th KNA election in 2012 and the 20th election in 2016 (for the 20th election, see Appendix).

Figures
Fig. 1. The effects of the administrative districts. The reference level, Seoul city, is colored gray in two figures. Its estimate is set as 0.
Fig. A.1. The effects of the administrative districts. The reference level, Seoul city, is colored gray in three figures. Its estimate is set as 0.
TABLES

### Table A.1

Predicted numbers of seats by three major broadcasting companies in Korea in the 20th election

KBS SBS MBC True
SNP (121, 143) (123, 147) (118, 136) 122
TMP (101, 123) (97, 120) (107, 128) 123
GMP (34, 41) (31, 43) (32, 42) 38

SNP = Saenuri Party; TMP = Minju Party; GMP = Gukmin Party.

### Table A.2

95% prediction interval of the number of seats

NoA MCind SB True
SNP (117.2, 134.1) (116.0, 135.0) (110.0, 159.0) 122
TMP (110.5, 127.7) (109.0, 129.0) (95.0, 141.0) 123
GMP (33.8, 40.4) (33.0, 41.0) (29.0, 41.0) 38

In the table “NoA” and “MCind” stand for the normal approximation and Monte Carlo approximation under independence assumption. “SB” stands for the spatial bootstrap procedure. SNP = Saenuri Party; TMP = Minju Party; GMP = Gukmin Party.

### Table A.3

Summary statistics for the spatial coefficients

party nhd-type est s.e. p-value
SNP ρwtn 0.065 0.053 0.217
ρbtw 0.189 0.039 <0.001

TMP ρwtn −0.013 0.054 0.803
ρbtw −0.198 0.045 <0.001

GMP ρwtn 0.013 0.076 0.868
ρbtw 0.145 0.081 0.075

“est” and “s.e.” are, respectively, an average and a standard deviation of B estimates from bootstrapped samples. P-values are calculated based on the normal distribution. SNP = Saenuri Party; TMP = Minju party; GMP = Gukmin party.

### Table 1

Prediction result from the exit poll data by three broadcasting systems in the 19th KNA election.

KBS SBS MBC True
SNP (131, 147) (126, 151) (130, 153) 152
MTP (131, 147) (128, 150) (128, 153) 127

KNA = Korean National Assembly; SNP = Saenuri Party; MTP = Minju Tonghap Party.

### Table 2

95% prediction interval of the number of seats

NoA MCind SB True
SNP (137.6, 152.4) (137.0, 153.0) (132.0, 158.0) 152
MTP (128.8, 142.9) (128.0, 144.0) (116.0, 142.0) 127

NoA = normal approximation; MCind = Monte Carlo approximation under independence assumption; SB = spatial bootstrap procedure; SNP = Saenuri Party; MTP = Minju Tonghap Party.

### Table 3

Summary statistics for the spatial coefficients

Party nhd-type est s.e. p-value
SNP ρwtn −0.057 0.055 0.297
ρbtw 0.092 0.092 0.321

MTP ρwtn −0.003 0.042 0.943
ρbtw −0.300 0.029 <0.001

“est” and “s.e.” are, respectively, an average and a standard deviation of B estimates from bootstrapped samples. P-values are calculated based on the normal distribution. SNP = Saenuri Party; MTP = Minju Tonghap Party.

### Table 4

95% prediction interval of the number of seats in small area

NoA MCind SB True
Belt-SNP (1.4, 5.7) (1.0, 6.0) (4.0, 8.0) 5
Belt-TMP (2.3, 6.6) (2.0, 7.0) (0.0, 4.0) 3
Seoul-SNP (10.5, 18.4) (10.0, 18.0) (9.0, 24.0) 16
Seoul-TMP (27.4, 35.2) (27.0, 35.0) (22.0, 37.0) 30

In the table “NoA” and “MCind” stand for the normal approximation and Monte Carlo approximation under independence assumption. “SB” stands for the spatial bootstrap procedure. “Belt-PARTY” refers to the outcome of PARTY in the Nakdonggang belt and “Seoul-PARTY” in Seoul city.

### Algorithm 1

Monte Carlo method with bootstrap sampling

 Input: {(pv, Xv), v ∈ V} 1: Estimate β0, β, ρ with the input as described in Section 2, and compute its residual ϵ based on equation (3.1). 2: forb = 1, . . . , B do 3: forv ∈ V do 4: Get a bootstrap sample $ϵv*$ by sampling uniformly from ${ϵv}v∈V$. 5: Obtain $zv*$ using equation (3.2) and transform back to success probabilities $pv*$. 6: Run the Bernoulli trial $Yv*$ with probability $pv*$. 7: end for 8: Compute their sum $T(b)=∑v∈VYv*$. 9: end for Output: {T(b): b = 1, . . . , B}: the bootstrap distribution of T.

References
1. Anselin L (1988). Spatial Econometrics, Dordrecht, Kluwer Academic Publishing.
2. Arnold M and Wied D (2010). Improved GMM estimation of the spatial autoregressive error model. Economic Letters, 108, 65-68.
3. Bafumi J, Erikson RS, and Wlezien C (2010). Ideological balancing, generic polls and midterm congressional elections. Journal of Politics, 72, 705-719.
4. Baltagi BH, Song SH, and Koh W (2003). Testing panel data regression models with spatial error correlation. Journal of Econometrics, 117, 123-150.
5. Brown P and Payne C (1975). Election night forecasting. Journal of the Royal Statistical Society, Series A, 138, 463-498.
6. Cochrane D and Orcutt GH (1949). Application of least squares regression to relationships containing auto-correlated Error Terms, 44 (245), 32-61.
7. Curtice J and Firth D (2008). Exit polling in a cold climate: the BBC-ITV experience in Britain in 2005. Journal of the Royal Statistical Society, Series A, 171, 509-539.
8. Greiner DJ and Quinn KM (2010). Exit polling and racial bloc voting: Combining individual-level and R × C ecological data. The Annals of Applied Statistics, 4, 1774-1796.
9. Hordijk L (1974). Spatial correlation in the disturbances of a linear interregional model. Regional and Urban Economics, 4, 117-140.
10. Huh MH (2008). Predicting major political parties’ number of seats in general election: the case of 2004 general election of Korea. Korean Association for Survey Research, 9, 87-100.
11. Kawk J and Choi B (2014). A comparison study for accuracy of exit poll based on nonresponse model. Journal of the Korean Data and Information Science Society, 25, 53-64.
12. Kwak ES, Kim JY, and Kim YW (2013). Analysis of forecasting error of the exit poll for the general election of 2012 in Korea. The Korean Association for Survey Research, 11, 33-55.
13. Lee L (2002). Consistency and efficiency of least squares estimation for mixed regressive, spatial autoregressive models. Econometric Theory, 18, 252-277.
14. Mendenhall W, Scheaffer RL, and Ott L (1971). Elementary Survey Sampling, Wadsworth Publishing Company.
15. Mitosfky WJ (1991). A Short History of Exit Polls, Sage, Newbury Park, CA.
16. Mitosfky WJ (1995). A Review of the 1992 VRS Exit Polls, Westview Press, Boulder, Colorado.
17. Ord K (1975). Estimation methods for models of spatial interaction. Journal of the American Statistical Association, 70, 120-126.
18. Wang W, Rothschild D, Goel S, and Gelman A (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31, 980-991.

format: plain


text:
# Why this condition in ODE?

1. Jan 3, 2006

### twoflower

Hi,

I just started playing with higher order ODEs and I'm stuck in one particular step. Here it is:

$$y^{''} + y = \frac{1}{\cos x}$$

1. step: I find fundamental solution system, which in this case is

$$[\cos x, \sin x]$$

So general solution looks like this:

$$y(x) = \alpha\cos x + \beta \sin x$$

Using the method of variation of parameters, $\alpha$ and $\beta$ become functions of x:

$$y(x) = \alpha(x)\cos x + \beta(x) \sin x$$

$$y'(x) = \alpha^{'}(x)\cos x - \alpha(x)\sin x + \beta^{'}(x) \sin x + \beta(x) \cos x$$

Now I don't understand the condition

$$\alpha^{'}(x)\cos x + \beta^{'}(x) \sin x = 0$$

Why does it have to be so?

Thanks for explanation!

2. Jan 3, 2006

### saltydog

Well you know it has to be something right? I mean:

$$\alpha^{'}(x)Cos(x)+\beta^{'}(x)Sin(x)=g(x)$$

Tell you what though, let's just make g(x) equal to the zero function and just see what happens. No harm in that right? I mean we're not talking asteriods or nothing? If we do, the math is certainly much easier when the second derivative is calculated and all the subsequent arithmetic is valid and we end up with a valid answer.

Works for me.

3. Jan 3, 2006

### twoflower

I still do not quite understand...What I thought is that we're finding for ONE particular solution, no mother which one of infinite number of them, so we FOR EXAMPLE, make sum of these derivatives equal to zero functions.

Is that it?

Anyway, I can't see whether it is really a correct step, I mean, we don't know it this sum really can be zero...you know what I mean.

4. Jan 3, 2006

### HallsofIvy

Staff Emeritus
Think how general this method is: any pair of functions can give any other function this way: If the two given functions are sin(x) and cos(x) and you want to get ex, just write
$$\frac{e^x}{2sin(x)}sin(x)+ \frac{e^x}{2cos(x)}cos(x). There are an infinite number of functions that will give you a solution. You are just "limiting the search" by requiring that [tex]\alpha'(x)sin(x)+ \beta'(x)cos(x)= 0$$. You use that particular requirement because that way, when you differentiate again, you wind up with a first order equation for $\alpha(x)$ and $\beta(x)$.

Here's an exercise: suppose you were third order equation that had solutions (to the homogeneous equation) $y_1(x),y_2(x), y_3(x)$ and we seek a solution of the form $y(x)= \alpha(x)y_1(x)+\beta(x)y_2(x)+\gamma(x)y_3(x)$. y'(x)= \alpha'(x)y_1(x)+\beta(x)'y_2(x)+\gamma'(x)y_3(x)+ \alpha(x)y_1'(x)+\beta(x)y_2'(x)+\gamma(x)y_3'(x)[/itex]. If we set
$\alpha'(x)y_1(x)+\beta'(x)y_2(x)+\gamma'(x)y_3(x)$ we are left with
$\alpha(x)y_1'(x)+\beta(x)y_2'(x)+\gamma(x)y_3'(x)$. Differentiating again, \alpha(x)'y_1'(x)+\beta'(x)y_2'(x)+\gamma'(x)y_3'(x)+ \alpha(x)y_1"(x)+\beta(x)y_2"(x)+\gamma(x)y_3"(x)[/itex]. What condition do we impose so that when we differentiate again (to get $y_1"'$, etc.) we still have only first derivatives of $\alpha(x)$, etc?

5. Jan 3, 2006

### twoflower

I see it, the condition is

$$\alpha(x)'y_1'(x)+\beta'(x)y_2'(x)+\gamma'(x)y_3'(x) = 0$$

Thank you HallsoftIvy!

Last edited: Jan 3, 2006


text:
# Beyond IIT #1

Algebra Level 2

determine sum of all possible non negative integral solutions of $$x^x=\frac{1}{\sqrt{2}}$$

×