text:
# Ways to Perform Calculations on Groups of Data in Tables4

Posted by Loren Shure,

Today I'd like to introduce a guest blogger, Stephen Doe, who works for the MATLAB Documentation team here at MathWorks. Stephen will discuss different methods for grouping data, and performing calculations on those groups, using Tables.

### Contents

#### Create Table

Tables are convenient containers for column-oriented data. The variables in a table can have different data types, but must have the same number of rows. You can access table data by row, by variable, or by variable name. And you can specify groups within table variables, to perform calculations on those groups. Today I will show several different ways to apply functions to groups in tables.

First, let's create a table. You can create a table using the table function, the Import Tool, or the readtable function. I'm going to use readtable to read data from a sample file that ships with MATLAB. The file outages.csv contains simulated data for electric power outages over a period of 12 years in the United States. Read the table. To display the first five rows, use table indexing.

T = readtable('outages.csv','Format','%C%D%f%f%D%C');
T(1:5,:)

ans =
5×6 table
Region         OutageTime        Loss     Customers     RestorationTime          Cause
_________    ________________    ______    __________    ________________    _______________
SouthWest    2002-02-01 12:18    458.98    1.8202e+06    2002-02-07 16:50    winter storm
SouthEast    2003-01-23 00:49    530.14    2.1204e+05    NaT                 winter storm
SouthEast    2003-02-07 21:15     289.4    1.4294e+05    2003-02-17 08:14    winter storm
West         2004-04-06 05:44    434.81    3.4037e+05    2004-04-06 06:10    equipment fault
MidWest      2002-03-16 06:18    186.44    2.1275e+05    2002-03-18 23:23    severe storm

There are six columns in the file. Since I want to specify the data types of the table variables, I use the Format name-value pair argument. T has two categorical variables (specified with %C), two datetime variables (%D) and two numeric variables (%f).

#### Perform Calculation on Each Numeric Variable

Once data is in a table, you can perform calculations on different table variables. One way is to use dot notation to refer to table variables by name and perform calculations. For example, calculate the total number of customers affected by power outages using the sum function, omitting any NaN values that might be in T.Customers.

totalCustomers = sum(T.Customers,'omitnan')

totalCustomers =
1.903e+08

You also can apply a function to every table variable, using the varfun function. varfun is particularly useful for applying a function and returning the results in another table.

However, you might want to apply a function only to variables of a particular type. You can specify the type of variable with the vartype function. Here I'm going to subscript into T for the numeric variables and calculate the mean of each one. And since the variables have some NaN values, I'm going to wrap mean in an anonymous function so I can use the 'omitnan' flag.

T2 = T(:,vartype('numeric'));
omean = @(x) mean(x,'omitnan');
meanT = varfun(omean,T2)

meanT =
1×2 table
Fun_Loss    Fun_Customers
________    _____________
563.89      1.6693e+05

#### Perform Calculation on Groups in Numeric Variables

A more interesting view of the table is to group the data, and start looking for differences between the groups. For example, what is the difference in the mean power loss between the Northeast and Southwest regions?

To answer this question, you can start by specifying one or more table variables as grouping variables. The grouping variables define groups into which you split the other table variables. Then you can apply a function to each group within each table variable and the results. For more on grouping variables, see Grouping Variables to Split Data.

You can use grouping variables with the varfun function I described above. Specify Region as the grouping variable in this table. It's a categorical variable, making it especially convenient to use as a grouping variable. Specify Loss and Customers as the input variables, and calculate the mean values by region for these two variables. I'm going to use the same omean function I specified above.

meanByRegion = varfun(omean,T,'GroupingVariables','Region',...
'InputVariables',{'Loss','Customers'})

meanByRegion =
5×4 table
Region      GroupCount    Fun_Loss    Fun_Customers
_________    __________    ________    _____________
MidWest      142           1137.7      2.4015e+05
NorthEast    557           551.65      1.4917e+05
SouthEast    389           495.35      1.6776e+05
SouthWest     26           493.88      2.6975e+05
West         354           433.37      1.5201e+05

Another way to apply a grouping variable to the data in a table variable is to use the accumarray function. If I convert T.Region to a numeric array, and extract T.Loss from T, then I can calculate mean power loss per region using accumarray, just as I did with varfun.

regions = double(T.Region);
A = accumarray(regions,T.Loss,[],omean)

A =
1137.7
551.65
495.35
493.88
433.37

accumarray operates on arrays. So to use accumarray on a table variable, you must extract it from the table. Also, the grouping variable must be numeric, and the output argument is an array, not a table. While you can use accumarray on table variables, it lacks many of the conveniences that varfun provides.

You can specify more than one grouping variable using varfun. For example, I'll specify Region and Cause as grouping variables. The groups are defined as combinations of regions and causes.

meanByRegionAndCause = varfun(omean,T,'GroupingVariables',{'Region','Cause'},...
'InputVariables',{'Loss','Customers'});
meanByRegionAndCause(1:5,:)

ans =
5×5 table
Region          Cause          GroupCount    Fun_Loss    Fun_Customers
_______    ________________    __________    ________    _____________
MidWest    attack              12                 0               0
MidWest    energy emergency    19            536.09           57603
MidWest    equipment fault      9            343.37           29704
MidWest    severe storm        31            1055.7      4.3584e+05
MidWest    thunder storm       32            941.07      1.3301e+05

#### Multiple Variables as Input Arguments to Function

Another simple calculation is the mean duration of the power outages by region. The basis of this calculation is the difference between the OutageTime and RestorationTime table variables.

You might think to use varfun for this calculation too. The difficulty is that varfun applies a function to each table variable separately. But this calculation requires both variables. In other words, we want to apply a function that requires two input arguments.

The solution is to use the rowfun function. With rowfun, all the table variables are taken as input arguments. If the table has N variables, then the function you apply must accept N input arguments.

To perform this calculation, I'll start by defining an anonymous function that subtracts one input argument from the other, and then calculates the mean. Then I will calculate the mean duration of power outages by region using rowfun.

meanDiff = @(a,b) mean(abs(a-b),'omitnan');
meanOutageTimes = rowfun(meanDiff,T,'GroupingVariables','Region',...
'InputVariables',{'OutageTime','RestorationTime'})

meanOutageTimes =
5×3 table
Region      GroupCount      Var3
_________    __________    _________
MidWest      142           819:14:44
NorthEast    557           581:02:18
SouthEast    389            40:49:49
SouthWest     26            59:31:07
West         354           673:27:12

Note that OutageTime and RestorationTime are datetime variables. You can perform arithmetic with datetime variables, just as you can with numeric variables.

To rename a variable, you can access the VariableNames property of the table and change its name there. Tables contain metadata, such as variable names and descriptions, in a property called Properties.

meanOutageTimes.Properties.VariableNames{'Var3'} = 'mean_OutageTime';
meanOutageTimes(1:5,:)

ans =
5×3 table
Region      GroupCount    mean_OutageTime
_________    __________    _______________
MidWest      142           819:14:44
NorthEast    557           581:02:18
SouthEast    389            40:49:49
SouthWest     26            59:31:07
West         354           673:27:12

#### Use Groups in Multiple Calculations and Tabulate Results

varfun and rowfun are suitable when you want to perform the same calculation with the table variables. But what if you want to calculate different quantities? For example, you might want to calculate the mean power loss, but also the minimum duration of power outages, by region. In that case, you can perform the calculations and tabulate the results using the findgroups and splitapply functions.

These functions work a little differently from varfun and rowfun. The findgroups function returns numeric indices that correspond to the grouping variables you specify. As a second output, it also returns a table of the groups. Then you can call splitapply to apply a function to each group within the variables.

For example, I'll calculate the mean power loss, the minimum outage time, and the maximum number of customers affected, by region. I will specify the groups once using findgroups. Then I can use the groups in multiple calls to splitapply.

I'm also going to switch gears a little bit, by removing all rows with NaN or missing values. I'll use the rmmissing function to remove these rows, so that I don't have to specify the 'omitnan' flag in every function call.

T = rmmissing(T);
[G,results] = findgroups(T(:,'Region'));
meanLoss = splitapply(@mean,T.Loss,G);
minOutageTime = splitapply(@min,T.RestorationTime - T.OutageTime,G);
maxCustomers = splitapply(@max,T.Customers,G)

maxCustomers =
3.295e+06
5.9689e+06
2.2249e+06
1.8202e+06
4.26e+06

The output of splitapply is an array. maxCustomers is a 5-by-1 numeric array. However, you can tabulate the results by adding them as variables to the table results, the second output from findgroups.

results.meanLoss = meanLoss;
results.minOutageTime = minOutageTime;
results.maxCustomers = maxCustomers

results =
5×4 table
Region      meanLoss    minOutageTime    maxCustomers
_________    ________    _____________    ____________
MidWest      907.19      00:00:00          3.295e+06
NorthEast    383.86      00:00:00         5.9689e+06
SouthEast    508.34      00:00:00         2.2249e+06
SouthWest    541.66      00:28:00         1.8202e+06
West         429.73      00:00:00           4.26e+06

#### Perform Calculations on Tall Table

So far, I've worked with a table that is small enough to fit into memory. But what if there is so much data that it won't all fit into memory? I might still want to treat it as a table and use the same functions.

That can easily be done by creating a tall table. A tall table is a kind of a tall array in MATLAB. Tall arrays and tall tables know how to read the data in one chunk at a time, perform the calculations you want, and then gather up the output at the end. The syntax for working with a tall table is very similar to that for working with a table.

For more information on tall arrays and tall tables, see Tall Arrays.

To show how this works, I'll create a tall table out of outages.csv. Of course it's a bit silly to create a tall table out of a file that fits in memory, but this illustrates how to work with a much larger table.

Then I'll call tall on ds, to create a tall table out of the datastore. Now I have a tall table that looks and acts very much like the table I worked with in the previous sections. One difference you see is that the tall table is displayed as an M-by-6 table, showing that the number of rows is not yet known.

ds = datastore('outages.csv');
T = tall(ds)

Starting parallel pool (parpool) using the 'local' profile ...
connected to 2 workers.
T =
M×6 tall table
Region          OutageTime        Loss     Customers     RestorationTime           Cause
___________    ________________    ______    __________    ________________    _________________
'SouthWest'    2002-02-01 12:18    458.98    1.8202e+06    2002-02-07 16:50    'winter storm'
'SouthEast'    2003-01-23 00:49    530.14    2.1204e+05    NaT                 'winter storm'
'SouthEast'    2003-02-07 21:15     289.4    1.4294e+05    2003-02-17 08:14    'winter storm'
'West'         2004-04-06 05:44    434.81    3.4037e+05    2004-04-06 06:10    'equipment fault'
'MidWest'      2002-03-16 06:18    186.44    2.1275e+05    2002-03-18 23:23    'severe storm'
'West'         2003-06-18 02:49         0             0    2003-06-18 10:54    'attack'
'West'         2004-06-20 14:39    231.29           NaN    2004-06-20 19:16    'equipment fault'
'West'         2002-06-06 19:28    311.86           NaN    2002-06-07 00:51    'equipment fault'
:              :                   :         :             :                   :
:              :                   :         :             :                   :

Now I can use findgroups and splitapply to perform some of the same calculations as before. Again I will wrap mean in an anonymous function so I can ignore NaN values, as I did above.

omean = @(x) mean(x,'omitnan');
T.Region = categorical(T.Region);
T.Cause = categorical(T.Cause);
G = findgroups(T.Region);
meanLoss = splitapply(omean,T.Loss,G);
meanCustomers = splitapply(omean,T.Customers,G)

meanCustomers =
M×1 tall double column vector
?
?
?
:
:

Instead of seeing the output, we see a trail of question marks. What does this mean? Simply that I have not performed any calculations yet. At this point, I have only defined the calculations I want to perform on the tall table. No calculations are performed until I gather the results, by calling the gather function. The output of gather is not a tall array and must fit into memory.

meanLoss = gather(meanLoss);
meanCustomers = gather(meanCustomers)

Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 4: Completed in 4 sec
- Pass 2 of 4: Completed in 2 sec
- Pass 3 of 4: Completed in 2 sec
- Pass 4 of 4: Completed in 2 sec
Evaluation completed in 25 sec
Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 1: Completed in 2 sec
Evaluation completed in 5 sec
meanCustomers =
2.4015e+05
1.4917e+05
1.6776e+05
2.6975e+05
1.5201e+05

Now let's tabulate the results in a table. Since meanLoss and meanCustomers are not tall arrays, results is not a tall table. Add the regions as another table variable.

results = table(meanLoss,meanCustomers)
region = unique(T.Region);
results.region = gather(region)

results =
5×2 table
meanLoss    meanCustomers
________    _____________
1137.7      2.4015e+05
551.65      1.4917e+05
495.35      1.6776e+05
493.88      2.6975e+05
433.37      1.5201e+05
Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 1: Completed in 1 sec
Evaluation completed in 2 sec
results =
5×3 table
meanLoss    meanCustomers     region
________    _____________    _________
1137.7      2.4015e+05       MidWest
551.65      1.4917e+05       NorthEast
495.35      1.6776e+05       SouthEast
493.88      2.6975e+05       SouthWest
433.37      1.5201e+05       West

#### Turning the Tables on You

Here I have shown you several ways you can group data in tables for calculations. To summarize, you can use one or more grouping variables to specify groups within the other table variables. Then you can apply functions to the groups in the table variables.

To apply:

• A function to a single table variable and return an array, use accumarray.
• The same function to each table variable and return a table, use varfun.
• A function that requires all the table variables as input arguments and return a table, use rowfun.
• Different functions to different table variables and build a table of results, use findgroups and splitapply.

How about you? Have done analysis with tables where accumarray, varfun, rowfun, findgroups and splitapply might have helped? Do you have cases where you need tall tables? Let us know here.

Get the MATLAB code

Published with MATLAB® R2017a


text:
# 4+3y=-9

## Simple and best practice solution for 4+3y=-9 equation. Check how easy it is, and learn it for the future. Our solution is simple, and easy to understand, so dont hesitate to use it as a solution of your homework.

If it's not what You are looking for type in the equation solver your own equation and let us solve it.

## Solution for 4+3y=-9 equation:

Simplifying
4 + 3y = -9
Solving
4 + 3y = -9
Solving for variable 'y'.
Move all terms containing y to the left, all other terms to the right.
Add '-4' to each side of the equation.
4 + -4 + 3y = -9 + -4
Combine like terms: 4 + -4 = 0
0 + 3y = -9 + -4
3y = -9 + -4
Combine like terms: -9 + -4 = -13
3y = -13
Divide each side by '3'.
y = -4.333333333
Simplifying
y = -4.333333333`

## Related pages

540-360factor 4x2 20x 253y 2x 96a squared9x2 6x 1antiderivative of sin 3xfraction percentage calculatorgcf of 84what is the prime factorization of 10011.9 percent as a decimal6x 2y 1032 qt to galsinx cosydecimals fractions percentsderivative of tanx 2what is 4xbln x 2 derivativee3x3.1415926what is 800 in roman numeralswhat is the gcf of 33 and 6635x25x2 3x 2how to put fractions on a calculatoraddition and subtraction fractions calculatorlcmcalculator6x 2x 15x 2y 10 graphwhat is the prime factorization of 1690.41666 as a fractionhow to solve i prty 9ywrite the prime factorization of 60tan15is234derivative of sin xy2cos 2xlcm of 6 8 12factor 3x 2-7x-6games 4u3decimals fractions and percentages1980 roman numerals9w 76gcf of 45 and 63what is the prime factorization of 248260-8factors of962cos 2x9999999999 numbergraph x 2y 5what is the lcm of 60 and 150greatest common monomial factor calculator4x graph3c and 5s2zxsecx cosx sinx4x-5y 0solve log 4x 10 3algebraic calculator with stepswhats prime factorization273-1002x2 6x 4roman numerals for 1974fractions multiplication calculatorxln xprime factorization of 630sin 3x derivativewhat is the prime factorization of 190p30xthe prime factorization of 175

format: plain


text:
• 学术研究 •

### 基于DT及PCA的DNN入侵检测模型

1. 天津大学 电气自动化与信息工程学院，天津 300072
• 出版日期:2021-08-01 发布日期:2021-08-02

### DNN Intrusion Detection Model Based on DT and PCA

WU Xiaodong, LIU Jinghao, JIN Jie, MAO Siping

1. School of Electrical Automation and Information Engineering, Tianjin University, Tianjin 300072, China
• Online:2021-08-01 Published:2021-08-02

Abstract:

Intrusion detection is an important field. The problems such as high false alarm rate, low detection rate, slow processing speed and high feature dimension plague the experts and scholars in this field. For those problems, this paper proposes an intrusion detection model DT-PCA-DNN combining DT (decision tree), PCA (principal com-ponent analysis) and DNN (deep neural networks) to improve the processing speed of the IDS (intrusion detection system) on the basis of a relatively high detection rate and a relatively low false alarm rate. In order to reduce the overall data volume and speed up the processing speed, DT is used to make a preliminary judgment on the data. The data judged as intrusion by DT are stored in a temporary sample set to optimize DT and DNN, and the data judged as normal are processed by PCA to reduce the data dimension and then processed by DNN for secondary judgment. If the DT structure is too deep, too much normal data will be judged as intrusion data. This will cause the subsequent DNN processing cannot effectively improve the overall accuracy, so DT uses a shallow structure. DNN uses the ReLU activation function that simplifies the calculation process of the neural network and the Adam optimization algorithm with faster convergence speed to speed up the data processing speed. According to the binary and multi-class classification experimental results on the NSL-KDD dataset, compared with other intrusion detection methods that use deep learning, this model, which achieves a relatively high detection rate and has a faster detection speed, solves the real-time problem of intrusion detection effectively.


text:
# Research

Published:

Published:

I am a postdoc at Stanford university, working under Chiara Sabatti. I got my PhD in biomathematics at UCLA, advised by Kenneth Lange and Janet Sinsheimer. Before that, I got a BA in mathematics at UC Berkeley.


text:
## Fake Critiques by Gill, Moldoveanu, and Weatherall Debunked

Foundations of physics and/or philosophy of physics, and in particular, posts on unresolved or controversial issues

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

gill1109 wrote:Certainly, the distinction between statistical independence, mathematical independence, and physical independence has confused readers and critics of Bell for all these long years, and still does. There are very interesting “metaphysical” analyses of when one kind of independence might imply another in recent works on Causality. I think of Judea Pearl’s “modern classic” (2nd edition) and of the new book on machine learning and causality by Jonas Peters et al, you can find a legal and free pdf on internet if you follow the links carefully. Both books even use the Bell-CHSH business as an example. They are both based on the modern theory of “graphical models” aka “Bayes nets”, a wonderful integration of computing, graphics, probability theory, and statistics. Lots and lots of money being made with them, too. I’ll try to add some links and exact references later.

I think these two books are pretty much essential reading for those interested in causality nowadays. Each includes the Bell-CHSH EPRB example as a short and elementary application of a much broader theory! Both have extensive discussions about causality in physics and about different kinds of independence and the relations between them.

Judea Pearl (2009) "Causality : Models, Reasoning, and Inference", *2nd edition*
http://bayes.cs.ucla.edu/BOOK-2K/
Ebook and paper both for sale, both worth every penny.

1st edition was (2000)

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf (2017), "Elements of Causal Inference - Foundations and Learning Algorithms".
https://mitpress.mit.edu/books/elements-causal-inference
Open access free pdf available. Paper version is worth every penny.

Judea Pearl's son Daniel was the journalist of the Wall Street Journal captured and murdered by the Taliban in 2002.
https://en.wikipedia.org/wiki/Daniel_Pearl

Jonas Peters works in algebraic statistics as well as in causality and in machine learning
http://web.math.ku.dk/~peters/

For completeness, I'd like to mention some "unconventional" books on the foundations of logic and mathematics and hence also about causality. The first is by the well-known Bell-denier Ilija Barukčić

Theoriae causalitatis principia mathematica
Ilija Barukčić

Prolegomena to the Complete Physical and Mathematical Theory of Rational Human Intelligence
Leo Depuydt
https://www.amazon.com/Prolegomena-Complete-Physical-Mathematical-Intelligence/dp/1618961012

A Logic of Exceptions
Thomas Colignatus / Thomas Cool
http://thomascool.eu/Papers/ALOE/Index.html
http://thomascool.eu/
gill1109
Mathematical Statistician

Posts: 1541
Joined: Tue Feb 04, 2014 10:39 pm
Location: Leiden

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Joy Christian wrote:***
Scott Aaronson seems to have responded to my refutation of his claims. The following comment was posted by him on his blog on March 13: https://www.scottaaronson.com/blog/?p=4145

Scott Aaronson wrote:
Once I slipped up, and made a single ad hominem comment about Cathy McGeoch (lead author of the “D-Wave machine gets a 3000x speedup” paper). I quickly apologized and I still regret it. Besides that, the one time in 13 years that I think I even came close to Bray and Pachter’s tone, was with the aggressive Bell’s-theorem-denialist Joy Christian. The reader can judge for herself whether Joy Christian and Manolis Kellis inhabit the same moral or intellectual universe. Even there, though, I regret getting drawn into the mud; it would’ve been more effective on my part to keep things professional.

Well, "regret" is cheap and does not undo the damage he has done to me (which is huge, if anyone cares to know). But it is good to know that he has made some progress: He now calls me "aggressive Bell's-theorem-denialist." That is progress because previously he used to call me a "Bell's-inequlity-denialist", which is not only false but also reveals his lack of understanding of the difference between Bell's inequality and Bell's theorem. How can anyone deny a mathematical inequality? On the other hand, Bell's theorem is not a theorem in the mathematical sense and its validity within physics can indeed be denied and have been from its very inception. So, again, it is good to know that Aaronson is finally making progress in learning some basic facts.

***

Dear Joy,

You and Fred seem fond of the following statement:

Joy Christian wrote:How can anyone deny a mathematical inequality?

Let me try. Since Bell's inequality (BI) was derived and offered -- and is still promoted -- in the context of the EPRB experiment, it is clearly false!

Mathematically, quantum mechanically and experimentally.

So I deny it -- not in the sense refusing to accept its existence -- but in the sense of refusing to admit its truth in its context.

Thanks.
Gordon Watson

Posts: 242
Joined: Wed Apr 30, 2014 4:39 am

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Gordon Watson wrote:So I deny it -- not in the sense refusing to accept its existence -- but in the sense of refusing to admit its truth in its context.
Thanks.

I am fond of it because I cannot but admit to its truth in its rightful context, and I recognize the enormous impact it should have on our thinking.

As I said before it is generally accepted as a pretty trivial theorem in computer science, more specifically, in distributed, classical, computing. It's been around for 60 years and been subjected to enormous scrutiny and animosity. It's not going to go away easily.

I am talking here about computer science as an abstract meta-science (or hypo, or hyper, depending on your taste), like pure mathematics itself. Or for that matter, like statistical science. Queen or handmaiden?

Gordon - your arguments are just plain wrong. You really need to learn some pure mathematics and some pure computer science.
gill1109
Mathematical Statistician

Posts: 1541
Joined: Tue Feb 04, 2014 10:39 pm
Location: Leiden

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

gill1109 wrote:
Gordon Watson wrote:So I deny it -- not in the sense refusing to accept its existence -- but in the sense of refusing to admit its truth in its context.
Thanks.

I am fond of it because I cannot but admit to its truth in its rightful context, and I recognize the enormous impact it should have on our thinking.

As I said before it is generally accepted as a pretty trivial theorem in computer science, more specifically, in distributed, classical, computing. It's been around for 60 years and been subjected to enormous scrutiny and animosity. It's not going to go away easily.

I am talking here about computer science as an abstract meta-science (or hypo, or hyper, depending on your taste), like pure mathematics itself. Or for that matter, like statistical science. Queen or handmaiden?

Gordon - your arguments are just plain wrong. You really need to learn some pure mathematics and some pure computer science.

Please: Enough of the verbage and avoidance.

One half-page of elementary algebra awaits your attention.

I am not aware of any simpler way for you [or Heinera] to be specific and show which of my arguments is wrong.
.
Gordon Watson

Posts: 242
Joined: Wed Apr 30, 2014 4:39 am

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Gordon Watson wrote:One half-page of elementary algebra awaits your attention. I am not aware of any simpler way for you [or Heinera] to be specific and show which of my arguments is wrong.
.

I’m interested in everyone else’s opinion of your half page of elementary algebra.

I don’t like algebra. I have a computer to do algebra for me, nowadays. I told you what’s wrong with your algebra, and you couldn’t, or wouldn’t, understand what I said. I wrote out a statement of Bell in the subjunctive mood which he should have used in the first place (but he was young and hasty, and used the indicative), and you didn’t see any difference.

Bell’s Theorem *is* less than a quarter of a page of utterly elementary probability theory. You are blind to it, Gordon. Worse still, you deliberately prefer to remain blind, rather than upset any of your preconceived ideas. You certainly aren’t going to waste any precious energy on learning something new. Whether about language or the foundations and philosophy of science. The biggest new developments in the field for 250 years. You call it verbiage and avoidance.

Pearl has even written popularising airport bookstore paperbacks for people who can’t find the time to read the real thing!
gill1109
Mathematical Statistician

Posts: 1541
Joined: Tue Feb 04, 2014 10:39 pm
Location: Leiden

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Well, Bell made a mistake that reduces his so-called theorem to a junk physics theory. The mistake is not really in his inequalities. It was in trying to compare QM and the quantum experiments to the inequalities. Now that we have successfully implemented Joy's hidden variable into quantum mechanics, Bell was certainly wrong about that. IOW, his interpretation of the actual physics was wrong. There are a whole bunch of bogus interpretations that are going to end up in the junk pile now. This is just the beginning.
FrediFizzx
Independent Physics Researcher

Posts: 1613
Joined: Tue Mar 19, 2013 7:12 pm
Location: N. California, USA

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

gill1109 wrote:
Gordon Watson wrote:One half-page of elementary algebra awaits your attention. I am not aware of any simpler way for you [or Heinera] to be specific and show which of my arguments is wrong.
.

I’m interested in everyone else’s opinion of your half page of elementary algebra.

I don’t like algebra. I have a computer to do algebra for me, nowadays. I told you what’s wrong with your algebra, and you couldn’t, or wouldn’t, understand what I said. I wrote out a statement of Bell in the subjunctive mood which he should have used in the first place (but he was young and hasty, and used the indicative), and you didn’t see any difference.

Bell’s Theorem *is* less than a quarter of a page of utterly elementary probability theory. You are blind to it, Gordon. Worse still, you deliberately prefer to remain blind, rather than upset any of your preconceived ideas. You certainly aren’t going to waste any precious energy on learning something new. Whether about language or the foundations and philosophy of science. The biggest new developments in the field for 250 years. You call it verbiage and avoidance.

Pearl has even written popularising airport bookstore paperbacks for people who can’t find the time to read the real thing!

Alas, you again miss the point. "The biggest new developments in the field for 250 years" was not the object of my comment.

That comment was directed at your current psycho-analytic writings. That's what I call verbiage and avoidance!

So please, returning to facts: please show me where "Bell’s Theorem *is* less than a quarter of a page of utterly elementary probability theory."

Since Bell carries -- to the end of his days and on the horns of a self-confessed dilemma -- this probabilistic expression

Bell locality: $P(AB|a,b,c,\lambda) =P(A|a,c,\lambda)P(|b,c,\lambda).$

why do you reject it but Bell did not?

PS: In the interest of brevity, further comment reserved for now.
.
Gordon Watson

Posts: 242
Joined: Wed Apr 30, 2014 4:39 am

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Gordon Watson wrote:
Bell locality: $P(AB|a,b,c,\lambda) =P(A|a,c,\lambda)P(|b,c,\lambda).$

Alone that formula does not mean Bell locality or anything. Only if you define all used symbols it may mean something.
Mikko

Posts: 152
Joined: Mon Feb 17, 2014 2:53 am

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Mikko wrote:
Gordon Watson wrote:
Bell locality: $P(AB|a,b,c,\lambda) =P(A|a,c,\lambda)P(B|b,c,\lambda).$ with GW edit

Alone that formula does not mean Bell locality or anything. Only if you define all used symbols it may mean something.

Mikko, I've corrected a typo.

When I introduced that eqn here -- for Richard Gill -- it was sourced to Norsen (on Norsen's terms), eqn (18): https://arxiv.org/pdf/quant-ph/0408105.pdf

"Bell locality" is also the term used by Norsen (and others).

HTH.
.
Gordon Watson

Posts: 242
Joined: Wed Apr 30, 2014 4:39 am

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

The paper by Travis Norsen was published in a conference proceedings or something like that with a paywall; obscure journal or organisation (not in my university library), so I can't access the final published version without spending my hard owned Euro's on it. More importantly, as far as I can see no one has ever cited that paper. The author's affiliation does not exactly boost confidence. Since I can't make sense of your writings why should I try to make sense of Norsen's?

Why don't you invite mr. Travis Norsen to join in the discussion here or on that google-group set up by Alexandre de Castro (devoted purely to Bell and quantum foundations). De Castro thinks that he has both a proof and a refutation of Bell's inequality, hence he can prove 0 = 1, hence there is a fundamental crisis in the foundations of science. We have to "re-think" and re-engineer both mathematics and logic. There are other smart people with the same opinion. Itamar Pitowsky (RIP) touched on this possibility; Han Geurdes is another; Ilija Barukčić yet another. Shouldn't your own logic force you down this same route?
gill1109
Mathematical Statistician

Posts: 1541
Joined: Tue Feb 04, 2014 10:39 pm
Location: Leiden

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Now, I don't see how anyone can refute the Bell inequalities. That is not the problem. But it is pretty easy to refute Bell's so-called theorem.
FrediFizzx
Independent Physics Researcher

Posts: 1613
Joined: Tue Mar 19, 2013 7:12 pm
Location: N. California, USA

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

FrediFizzx wrote:Now, I don't see how anyone can refute the Bell inequalities. That is not the problem. But it is pretty easy to refute Bell's so-called theorem.

Exactly. The Bell inequalities are pretty trivial. [I don't think it makes sense to accuse Bell of *stealing* them from Boole. Not many people have read Boole from cover to cover, and Boole has the inequalities as an exercise for the reader; he doesn't give the solution, he didn't publish an instructors manual. It was nice that Itamar Pitowsky discovered/uncovered them...]

Meta-physical implications of the inequalities - that is the interesting thing! We are discussing them now and we have been discussing them since they were first published. Fred, what you think easy to refute is the commonly accepted "executive summary" that quantum mechanics is non-local, or even, supposing QM predictions are more or less right, that nature is non-local.
gill1109
Mathematical Statistician

Posts: 1541
Joined: Tue Feb 04, 2014 10:39 pm
Location: Leiden

### Re: Fake Critiques by Gill, Moldoveanu, and Weatherall Debun

Gordon Watson wrote:
Mikko wrote:
Gordon Watson wrote:
Bell locality: $P(AB|a,b,c,\lambda) =P(A|a,c,\lambda)P(B|b,c,\lambda).$ with GW edit

Alone that formula does not mean Bell locality or anything. Only if you define all used symbols it may mean something.

Mikko, I've corrected a typo.

When I introduced that eqn here -- for Richard Gill -- it was sourced to Norsen (on Norsen's terms), eqn (18): https://arxiv.org/pdf/quant-ph/0408105.pdf

"Bell locality" is also the term used by Norsen (and others).

HTH.

When that equation was introduced there was no reference to Norsen. So you should explain the symbols, especially "c" which is not present in Norsen's equation 18 and which seems to have the same role as λ. Norsen's equation (18) is of course a good way to express locality:
P(A,B|a,b,λ) = P(A|a,λ) P(B|b,λ),
where P is probability and must not be confused with Bell's P. Other symbols are as in Bell's paper except that Norsen has a circumflex on a and b whereas Bell has an arrow. But the meaning is the same: they are directions.
Mikko

Posts: 152
Joined: Mon Feb 17, 2014 2:53 am

Previous


text:
Thread: On Leyland Primes View Single Post
2015-02-11, 15:24   #8
davar55

May 2004
New York City

2·32·5·47 Posts

Quote:
 Originally Posted by davar55 In any case, I was looking for two different representations of a prime p as Leyland-like primes, not an x,y that produced two different primes (which is a valid additional question, so thanks for an example.)
This was what I should have asked in the OP, without the labelled packaging.


text:
| Home | Journals | Books | Paper submission | About us and our mission | News | Contact us |

Site Search:

## Online Edition, Vol. 6, 2012, no. 21-24

N. Ahmed, D. Sarma, H. Deka
MHD mixed convection and mass transfer from an infinite vertical porous plate with chemical reaction in presence of a heat source
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1011-1020.

C. Duraisamy, M. Dhavamani
Intuitionistic non-continuous functions
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1021-1029.

F. Yusof, Foo Hui-Mean
Use of statistical distribution for drought analysis
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1031-1051.

D. Arivudainambi, V. Poongothai
A feedback control model with negligible switching costs for a retail facility
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1053-1070.

D. Arivudainambi, P. Godhandaraman
A batch arrival retrial queue with two phases of service, feedback and K optional vacations
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1071-1087.

S. Halici
On some Fibonacci-type polynomials
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1089-1093.

K. Teerapabolarn
A pointwise approximation of generalized binomial by Poisson distribution
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1095-1104.

S. Louvros, A. C. Iossifides
A general distribution approximation method for mobile network traffic modeling
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1105-1112.

Synchronization - like behaviors of dynamical systems with non negative Lyapunov exponents
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1113-1120.

M. K. Meher, G. C. Panda, S. K. Sahu
An inventory model with Weibull deterioration rate under the delay in payment in demand decling market
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1121-1133.

M. Saraj, N. Safaei
Solving bi-level programming problems on using global criterion method with an interval approach
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1135-1141.

N. I. Rashwan, M. El dereny
The comparison between results of application Bayesian and maximum likelihood approaches on logistic regression model for prostate cancer data
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1143-1158.

G. Ujjainkar, V. K. Gupta, B. Singh, R. Khandelwal, N. Trivedi
An epidemic model with modified non-monotonic incidence rate under treatment
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1159-1171.

N. Murugesan, O. V. Shanmuga Sundaram
Construction of state diagram of regular expressions using derivatives
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1173-1179.

O. Bumbariu, V. Berinde
An empirical study of the \epsilon-algorithm for accelerating numerical sequences
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1181-1190.

F. Cherif
On the spectra of almost periodic symmetric positive definite matrices functions
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1191-1197.

Xiu-qin Lou, Li-min Liu
Utility indifference pricing and hedging of Stein-Stein model
Appl. Math. Sci., Vol. 6, 2012, no. 21-24, 1199-1204.


text:
# Which one of the following metal oxides is antiferromagnetic in nature

1. A

${\mathrm{MnO}}_{2}$

2. B

${\mathrm{TiO}}_{2}$

3. C

${\mathrm{VO}}_{2}$

4. D

${\mathrm{CrO}}_{2}$

FREE Lve Classes, PDFs, Solved Questions, PYQ's, Mock Tests, Practice Tests, and Test Series!

+91

Verify OTP Code (required)

### Solution:

Antiferromagnetic substance are those substance whose spins are aligned in such a way that the total net moment will be zero. Spins of antiferromagnetic compounds are aligned in opposite direction with respect to the neighboring spin.

${\mathrm{MnO}}_{2}$ is antiferromagnetic.

## Related content

Join Infinity Learn Regular Class Program!


text:
[MiKTeX] miktex 2.9 Issue with \hat

 [MiKTeX] miktex 2.9 Issue with \hat From: Michael Olson - 2012-09-10 18:36:04 Why do $\hat{ \theta}$, $\hat{ \beta}$ and $\hat{ \sigma}$ work as expected but $\hat{ \alpha}$ fails? Mike Olson, Ph.D. Analyst Tiber Creek Consulting (703) 766-2150 x6471 michael.olson@...

 [MiKTeX] miktex 2.9 Issue with \hat From: Michael Olson - 2012-09-10 18:36:04 Why do $\hat{ \theta}$, $\hat{ \beta}$ and $\hat{ \sigma}$ work as expected but $\hat{ \alpha}$ fails? Mike Olson, Ph.D. Analyst Tiber Creek Consulting (703) 766-2150 x6471 michael.olson@...
 Re: [MiKTeX] miktex 2.9 Issue with \hat From: Ariovaldo V Garcia - 2012-09-10 18:48:35 Hi, What do you mean by "it fails"? For me it works as expected. Ariovaldo On Mon, Sep 10, 2012 at 3:20 PM, Michael Olson wrote: > Why do $\hat{ \theta}$, $\hat{ \beta}$ and $\hat{ \sigma}$ work as > expected but $\hat{ \alpha}$ fails? > > Mike Olson, Ph.D. > Analyst > Tiber Creek Consulting > (703) 766-2150 x6471 > michael.olson@... > >
 Re: [MiKTeX] miktex 2.9 Issue with \hat From: drmathprog - 2012-09-10 20:19:20 I'm running MikTex 2.9 within WinEdt, so it's possible that is a factor. When I use $\hat{\alpha}$, I get a console output that show: ! Missing $inserted.$ l.22 ... $\hat{\alpha}$ ? Process has been terminated ... When I use $\hat{\beta}$ or any other greek alphabetical symbol, I get the proper expected result. Only alpha seems to cause this result. Michael Olson-7 wrote: > > Why do $\hat{ \theta}$, $\hat{ \beta}$ and $\hat{ \sigma}$ work as > expected but $\hat{ \alpha}$ fails? > > Mike Olson, Ph.D. > Analyst > Tiber Creek Consulting > (703) 766-2150 x6471 > michael.olson@... > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > MiKTeX-Users mailing list > MiKTeX-Users@... > https://lists.sourceforge.net/lists/listinfo/miktex-users > > -- View this message in context: http://old.nabble.com/miktex-2.9-Issue-with-%5Chat-tp34414658p34415167.html Sent from the miktex-users mailing list archive at Nabble.com.
 Re: [MiKTeX] miktex 2.9 Issue with \hat From: Ariovaldo V Garcia - 2012-09-10 20:55:46 I am running MikTex 2.8 and WinEdt 5 and, as I wrote, it works fine. Ariovaldo On Mon, Sep 10, 2012 at 5:19 PM, drmathprog wrote: > > I'm running MikTex 2.9 within WinEdt, so it's possible that is a factor. > > When I use $\hat{\alpha}$, I get a console output that show: > > ! Missing $inserted. > >$ > l.22 ... $\hat{\alpha} >$ > ? > > Process has been terminated ... > > > When I use $\hat{\beta}$ or any other greek alphabetical symbol, I get the > proper expected result. Only alpha seems to cause this result. > > > > Michael Olson-7 wrote: > > > > Why do $\hat{ \theta}$, $\hat{ \beta}$ and $\hat{ \sigma}$ work as > > expected but $\hat{ \alpha}$ fails? > > > > Mike Olson, Ph.D. > > Analyst > > Tiber Creek Consulting > > (703) 766-2150 x6471 > > michael.olson@... > > > > > ------------------------------------------------------------------------------ > > Live Security Virtual Conference > > Exclusive live event will cover all the ways today's security and > > threat landscape has changed and how IT managers can respond. Discussions > > will include endpoint security, mobile security and the latest in malware > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > _______________________________________________ > > MiKTeX-Users mailing list > > MiKTeX-Users@... > > https://lists.sourceforge.net/lists/listinfo/miktex-users > > > > > > -- > View this message in context: > http://old.nabble.com/miktex-2.9-Issue-with-%5Chat-tp34414658p34415167.html > Sent from the miktex-users mailing list archive at Nabble.com. > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > MiKTeX-Users mailing list > MiKTeX-Users@... > https://lists.sourceforge.net/lists/listinfo/miktex-users >
 Re: [MiKTeX] miktex 2.9 Issue with \hat From: Alan G Isaac - 2012-09-10 22:14:04 On 9/10/2012 4:19 PM, drmathprog wrote: > When I use $\hat{\alpha}$, I get a console output that show: > > ! Missing $inserted. > >$ > l.22 ... $\hat{\alpha} >$ > ? > > Process has been terminated ... > > > When I use $\hat{\beta}$ or any other greek alphabetical symbol, I get the > proper expected result. Highly unlikely. Please provide a minimum document that produces the problem. Most likely, you have an odd number of \$. Try this: go to your line 22 and replace alpha with beta, with no other changes, and try compiling. Alan Isaac


text:
adjusting symbols above letter in mathmode

I know there are plenty of similar questions. If you check out the lower picture, you can see that i've tried a bunch of things to solve my problem. That's just one picture...i'm kind of obsessed with the detail here. I've also tried all \smallfrowns and \arrowright commands and they are even uglier. The biggest problem seems to be that the curve above the letter is not proportional to the size of the letter...AND most of the times it's to far away (height). I deleted my solution so please do not ask what commands i have used in my example. That was weeks ago. But if you need some sort of MWE let me know.

However, what i want is a solution like in the first picture.

My documentclass: \documentclass{scrreprt}

in case it makes a difference, i also use: \usepackage{amsmath, amssymb}

Here I offer two variants, where a right paren (rotated) is scaled to the proper width, or where it is stretched to the proper width. The vertical offset is controlled by the optional argument to \stackon. I personally find the \stretchto option superior.

EDIT: Definitions tweaked to auto-handle smaller mathstyles.

\documentclass{article}
\usepackage{stackengine,scalerel,graphicx}
\stackMath
\newcommand\overarc[1]{\ThisStyle{%
\setbox0=\hbox{$\SavedStyle#1$}%
\stackon[.5pt]{\SavedStyle#1}{%
\rotatebox{90}{$\SavedStyle\scaleto{)}{.95\wd0}$}}}}
\begin{document}
\centering
Using \verb|\scaleto|:
$\overarc{x} \quad \overarc{X} \quad \overarc{ABC}$
$\frac{F_z}{\overarc{z}} \frac{F_z}{\scriptstyle\overarc{z}} \frac{F_z}{\scriptscriptstyle\overarc{z}}$

\renewcommand\overarc[1]{\ThisStyle{%
\setbox0=\hbox{$\SavedStyle#1$}%
\stackon[.5pt]{\SavedStyle#1}{%
\rotatebox{90}{$\SavedStyle\stretchto{)}{.95\wd0}$}}}}
Using \verb|\stretchto|:
$\overarc{x} \quad \overarc{X} \quad \overarc{ABC}$
$\frac{F_z}{\overarc{z}} \frac{F_z}{\scriptstyle\overarc{z}} \frac{F_z}{\scriptscriptstyle\overarc{z}}$
\end{document}

• @goodstuff Note that this formulation should automatically work in \scriptstyle as well as \scriptscriptstyle. – Steven B. Segletes Dec 8 '14 at 16:45
• do you mean i could have used \smallfrown with \scriptstyle to get the adjustment i wanted? i don't really get what you mean. – goodstuff Dec 8 '14 at 17:22
• @goodstuff Please see my edit. I needed a small tweak to the definitions, but show how the smaller math styles are handled automatically. – Steven B. Segletes Dec 8 '14 at 17:46


text:
# Box topology not first countable (constructive proof)

Let $(\mathbb{R}^\omega,\tau)$ be the box topology space. I have found that everyone proofs that this topology is not first countable by finding a contradiction, but I thought of a constructive way. Would appreciate comments if I'm missing something. Goes like this:

Let $x\in\mathbb{R}^\omega$ and let $B(x)$ be a local basis for $x$. Then take $a_{i\varepsilon}=x_i-\varepsilon$ and $b_{i\varepsilon}=x_i+\varepsilon$, and let $$I_\varepsilon=\prod_{i\in\mathbb{N}}\, (a_{i\varepsilon},b_{i\varepsilon})$$ So, in particular, we get that $\forall\varepsilon\in (0,1),\,\,$ $I_\varepsilon$ are neighbourhoods of $x$. So $\forall\varepsilon\in (0,1)$ $\exists B_\varepsilon\in B(x)$ s.t. $x\in B_\varepsilon\subseteq I_\varepsilon$. Thus there is at least a continnum of elements in $B(x)$, i.e., $|B(x)|\ge c$ and thereby $\mathbb{R}^\omega$ is not first countable.

Can you make the same argument for $\mathbb{R}$? $\mathbb{R}$ is first countable, of course.

The point is you can say essentially the same words, so there is a mistake in your argument.

Hint:

You implicitly claim a map is injective, but it isn't.

Identify $\mathbb R^{\omega}$ with the set of $f:\omega \to \mathbb R.$ A local base at $f$ is $B_f=\{B(f,g)| g:\omega \to \mathbb R^+\}$ where $B(f,g)=\{h\in \mathbb R^{\omega}\;|\;\forall n \in \omega\;(\;|f(n)-h(n)|<g(n)\;)\}.$

We can show that any local base $C_f$ at $f$ is uncountable as follows: For each $c\in C_f$ let $c\supset B(f,g_c)\in B_f.$ Now if $\{c(j)\;|\;j\in\omega\}\subset C,$ then for $n\in \omega$ let $G(n)=\frac {1}{2}\min \{g_{c(j)}(n)\;|\;j\leq n\}.$

No member of $\{c(n):n\in \omega\}$ is a subset of $B(f,G)$ because for $n\in \omega$ we have $$f+\frac {3}{4}g_{c(n)}\in c(n) \backslash B(f,G)$$.....because $f+\frac {3}{4}g_{c(n)}\in B(f,g_{c(n)})\subset c(n),$ but $\frac {3}{4}g_{c(n)}(n)>G(n).$

So no countable subset of $C$ is a local base at $f.$


text: Abstract—A very-high radix algorithm and implementation for circular CORDIC is presented. We first present in depth the algorithm for the vectoring mode in which the selection of the digits is performed by rounding of the control variable. To assure convergence with this kind of selection, the operands are prescaled. However, in the CORDIC algorithm, the coordinate <tmath>$x$</tmath> varies during the execution so several scalings might be needed; we show that two scalings are sufficient. Moreover, the compensation of the variable scale factor (including the CORDIC scale factor and the prescaling factors) is done by computing the logarithm of the scale factor and performing the compensation by an exponential. Then, we combine, in a unified unit, the proposed vectoring algorithm and the very-high radix rotation algorithm, which was previously proposed by the authors. We compare with low-radix implementations in terms of latency and hardware complexity. Estimations of the delay for 32-bit precision show a speedup of about two with respect to the radix-4 case with redundant addition. This speedup is obtained at the cost of an increase in the hardware complexity, which is moderate for the pipelined implementation. We also compare at the algorithmic level with other very-high radix proposals, demonstrating the advantages of our algorithms.


text:
Original article Clinical endoscopy: Editorial| Volume 92, ISSUE 4, P912-913, October 01, 2020

# Artificial intelligence for polyp characterization: Don’t save on your competence!

#### Abbreviations:

AI (artificial intelligence), CRC (colorectal cancer)
Colonoscopy screening comes with a significant cost. If you sum up the costs due to health care personnel, endoscopy technology, facilities, and histopathology, colonoscopy is among the most expensive diagnostic procedures, with estimates comparable with whole-body CT or magnetic resonance imaging. When these costs are projected at population level, the absolute magnitude is exceptional, inasmuch as it is estimated to correspond to an annual gross expenditure of more than U.S. $775 million. • Mori Y. • Kudo S.-e. • East J.E. • et al. Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video). It could be argued that this is unfair. Colonoscopy represents one of the most effective preventive interventions. To prevent or downstage colorectal cancer (CRC) also means to save money, namely, the expenses of those involved in surgery, chemoradiotherapy, and palliative treatment. Isn’t it true that all the savings in CRC-related costs are enough to offset all the direct and indirect cost of screening colonoscopy? • Zhong G.-C. • Sun W.-P. • Wan L. • et al. Efficacy and cost-effectiveness of fecal immunochemical test versus colonoscopy in colorectal cancer screening: a systematic review and meta-analysis. Unexpectedly, what appeared to be a costly procedure is suddenly converted to one of the most cost-effective preventive strategies! There is, however, a temporal window between the costs and the savings. Although costs are to be paid now, the savings will be delayed by 10 to 20 years because of the slow natural history of the adenoma-carcinoma sequence. Costs are certain, but the savings may be shadowed by unexpected factors, such as death by competing causes and/or incidence of postcolonoscopy CRC resulting from a missed lesion or incomplete resection. This is why the long-term benefit of colonoscopy must be discounted because “a bird in the hand is worth 2 in the bush.” Optical diagnosis of diminutive polyps remains by far the most attractive intervention for an immediate saving on the costs of screening colonoscopy. Because of their high prevalence, these lesions disproportionately account for most of the histopathology costs, representing nearly 10% of the whole colonoscopy cost in the United States. • Mori Y. • Kudo S.-e. • East J.E. • et al. Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video). In addition, the pathologic relevance is questionable inasmuch as it is mostly represented by either nonadvanced adenomas or indolent hyperplastic polyps. In addition, the role of pathology as reference standard has been questioned because of the possibility of inadequate orientation or insufficient sampling of the lesion. • Shahidi N. • Rex D.K. • Kaltenbach T. • et al. Use of endoscopic impression, artificial intelligence, and pathologist interpretation to resolve discrepancies between endoscopy and pathology analyses of diminutive colorectal polyps. Endoscopy classifications, based on the use of blue-light imaging, showed a high accuracy in the in-vivo prediction of histologic diagnosis, generating cost-saving strategies like “leave in situ” and “resect and discard.” Disappointingly, their implementation in a community setting failed because of their lower than expected accuracy and excessive interoperator variability. • Wang L.M. • East J.E. Diminutive polyp cancers and the DISCARD strategy: Much ado about nothing or the end of the affair?. Will artificial intelligence (AI) revitalize the expected savings from optical diagnosis? By modeling the results of a breakthrough trial, Mori et al • Mori Y. • Kudo S.-e. • East J.E. • et al. Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video). estimated in$85 million the potential annual savings of using the high accuracy of EndoBRAIN, a machine-learning system compliant with the criteria of Preservation and Incorporation of Valuable Endoscopic Innovation II, to implement a leave-in-situ strategy for rectosigmoid hyperplastic polyps ≤5 mm.
• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
The value of this estimate is in the new opportunity to replicate an adequate standard of optical diagnosis at the community level, irrespectively of the level of experience. This was nicely shown in the original trial, where the performance of EndoBRAIN was similar between expert and nonexpert endoscopists. The fact that Mori et al
• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
limited their financial analysis to the leave-in-situ strategy also increases the reliability of their estimate, because it is by far more endoscopist friendly than the resect-and-discard strategy. Third, when the $125 saving per colonoscopy is combined with a negative predictive value of 95.2%, the tradeoff would be a convincing saving of over$5000 per 1 adenoma misclassified as hyperplastic.
• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
Will such savings be offset by the costs of AI implementation? In comparison with previous innovations in the field of optical diagnosis, such as advanced endoscopy imaging or magnification, the estimated AI cost of \$10 per colonoscopy is negligible. This was somewhat expected when it is considered that AI systems are simple real-time software that do not require any adaptations of the original endoscopic image. Of note, the initial investment in AI appeared to be completely compensated by its savings in pathology costs, further supporting a widespread implementation of such technology.
• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
It could be argued that the limited availability of endocytoscope, adopted by Mori et al,
• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
would prevent the generalizability of such an estimate. However, as is also fairly suggested by the same authors, accuracy values similar to those shown by EndoBRAIN have been reproduced with deep learning systems that are easily adapted to nonmagnified endoscopes.
• Byrne M.F.
• Soudan F.
• et al.
Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model.
Will AI-triggered savings suddenly eliminate all the barriers preventing optical diagnosis to be used in clinical practice? Differently from detection, the use of AI for polyp characterization implies a higher cognitive skill possessed by the endoscopist. In the case of detection, the endoscopist, irrespective of the level of expertise, will consistently confirm or not the lesion flagged by AI with a high level of confidence, as simply related to a morphologic or color change of the mucosa. By contrast, only an endoscopist competent in optical diagnosis will be able to accept or refuse the AI characterization with a high level of confidence based on a complex gestaltic analysis of the surface and vessel features. Nonexpert endoscopists will passively exploit the AI prediction without any chance to challenge it with a high degree of confidence, generating the risk of a higher level of automation of the diagnostic process. If the choice is between a diagnosis by an expert pathologist and an automatic AI prediction merely certified by an inexpert endoscopist, health systems will remain reluctant to implement the leave-in-situ strategy, irrespective of the magnitude of the financial savings. Who will be finally responsible for an incorrect diagnosis: the endoscopist, the software developer, the health systems, or all of them together?
Only a rigorously sequential algorithm between AI prediction and endoscopist’s confidence may fasten its incorporation into clinical practice. The leave-in-situ strategy should be limited to polyps that are both predicted as hyperplastic by AI and confirmed with a high level of confidence by competent endoscopists. In the case of any discrepancy or low level of confidence, histologic analysis should be required. This argues against using AI accuracy to bypass a suboptimal competence in optical diagnosis. The formal recommendations that “the optical diagnosis…can be performed only by experienced endoscopists who are adequately trained and audited” is not challenged by AI.
• Bisschops R.
• East J.E.
• Hassan C.
• et al.
Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) Guideline - Update 2019.
Competence in optical diagnosis, acquired with a structured curriculum,
• Bisschops R.
• East J.E.
• Hassan C.
• et al.
Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) Guideline - Update 2019.
is the prerequisite, not the final outcome of AI implementation!
In conclusion, the breakthrough clinical benefit coming from AI-based polyp detection and characterization may be expected to be fully supported by the saving in useless polypectomies, pathologic examinations, and prevention of postcolonoscopy CRC. However, this must trigger a timely improvement of the endoscopists’ standard, especially in polyp characterization, because what can be self-paying will never be accepted to be also self-driving.

## Disclosure

Dr Hassan and Dr Repici are consultants for Fujifilm and Medtronic. The other author disclosed no financial relationships.

## References

• Mori Y.
• Kudo S.-e.
• East J.E.
• et al.
Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video).
Gastrointest Endosc. 2020; 92: 905-911.e1
• Zhong G.-C.
• Sun W.-P.
• Wan L.
• et al.
Efficacy and cost-effectiveness of fecal immunochemical test versus colonoscopy in colorectal cancer screening: a systematic review and meta-analysis.
Gastrointest Endosc. 2020; 91: 684-697.e15
• Shahidi N.
• Rex D.K.
• Kaltenbach T.
• et al.
Use of endoscopic impression, artificial intelligence, and pathologist interpretation to resolve discrepancies between endoscopy and pathology analyses of diminutive colorectal polyps.
Gastroenterology. 2020; 158: 783-785.e1
• Wang L.M.
• East J.E.
Diminutive polyp cancers and the DISCARD strategy: Much ado about nothing or the end of the affair?.
Gastrointest Endosc. 2015; 82: 385-388
• Byrne M.F.


text:
Opened 3 years ago

Closed 3 years ago

Project list

Reported by: Owned by: endquote falkb normal SimpleMultiProjectPlugin normal 1.0

Description

Maybe this is in there somewhere already, but -- I would like the front page of my trac instance to include a list of projects that the current user has access to, linked to a report of open tickets for that project. Seems like this would be a great feature addition.

comment:1 Changed 3 years ago by anonymous

Not sure if that's what you're thinking of, but how about the TicketQuery macro? E.g.:

[[TicketQuery(project=Foo1,status!=closed)]]
[[TicketQuery(project=Foo2,status!=closed)]]

See here for more details, like about displaying options.

comment:2 Changed 3 years ago by endquote

That's interesting, but not really what I'm thinking of... I think what I want is a macro like:

[ProjectList(href=/query?project=%project%)]

Which would output something like:

Project1

Project2

Project3

Of course I could create this list manually without too much trouble, but it would be nice to have it automatically update when a project is added or closed.

comment:3 Changed 3 years ago by anonymous

This gets you a bit closer: [[TicketQuery(group=project,format=progress)]]

comment:4 Changed 3 years ago by endquote

That is pretty cool, but would still require me to add a new TicketQuery for each project when the project is added. I guess what I'm really looking for is a way to loop through projects.

[ProjectList(orderby=name,closed=false)]
[[TicketQuery(group=project,format=progress)]]
[/ProjectList]

Or:

[ProjectList(orderby=name,closed=false)]

[%project.name%]: [%project.description%]

[/ProjectList]

I know the wiki syntax is all wrong on that, but that's the general idea.

No big deal though, it's not as if I'm adding projects so often.

comment:5 Changed 3 years ago by endquote

Something like this could work too:

[[TicketQuery(group=project,format=progress)]]

However this gives me an error...

comment:6 follow-up:  7 Changed 3 years ago by falkb

Hmm... I noticed the parameter group cannot have a value which is a custom ticket field. group=project doesn't work but it works with e.g. group=milestone. Seems like a restriction of the TicketQuery macro.

[[TicketQuery(group=milestone,project=~Project&status!=closed, format=progress)]]

It lists all milestones of projects containing the pattern 'Project' as progress bars which is also 'Project1', 'Project2' and 'Project3'.

Generally, I'd rather try to find a solution with TicketQuery. You know bloating a plugin too much is the beginning of its death.

comment:7 in reply to:  6 Changed 3 years ago by falkb

[[TicketQuery(group=milestone,project=~Project&status!=closed, format=progress)]]

I wished I could set a parameter for showing only milestones with a set date.

comment:8 Changed 3 years ago by endquote

Should we log a ticket against TicketQuery to support grouping by custom fields? Guess I'll just do it manually in the meantime.

comment:9 Changed 3 years ago by endquote

Maybe this could do it: SqlQueryMacro

Last edited 3 years ago by Ryan J Ollos (previous) (diff)

comment:10 follow-up:  11 Changed 3 years ago by endquote

I was able to get WikiTableMacro to work ok. It's not exactly what I want (the roadmap view grouped by project would be super cool), but this'll do for now. Proper support within TicketQuery would be best.

SQLTable
SELECT '[/query?status=accepted&status=assigned&status=new&status=reopened&project=' || name || ' ' || name || ' — ' || summary || ']' as Projects FROM smp_project

Last edited 3 years ago by endquote (previous) (diff)

comment:11 in reply to:  10 Changed 3 years ago by falkb

Resolution: → wontfix new → closed

I was able to get WikiTableMacro to work ok. It's not exactly what I want (the roadmap view grouped by project would be super cool), but this'll do for now. Proper support within TicketQuery would be best.

Don't hesitate to file a bug report or enhancement ticket, respectively.

SQLTable
SELECT '[/query?status=accepted&status=assigned&status=new&status=reopened&project=' || name || ' ' || name || ' — ' || summary || ']' as Projects FROM smp_project

Cool solution. Glad you've found something useful. Maybe it's a good idea to filter by project completion state also (column 'closed' in smp_project).

The ticket query suggestion from comment:6 doesn't fit your needs, does it? Here, we live from milestone to milestone (or version to version) anyway. So that would be useful here.

After all IMHO it's definitely up to the query system outside this plugin to feature the things reported here.

comment:12 Changed 3 years ago by endquote

I logged an enhancement request at trac: http://trac.edgewall.org/ticket/11479

Thanks for the ideas!

comment:13 Changed 3 years ago by endquote

My current solution here works pretty well...

#! SQLTable
SELECT '[/query?status=accepted&status=assigned&status=new&status=reopened&project=' || name || ' ' || name || ' — ' || summary || ']' as Projects FROM smp_project WHERE closed = 0 ORDER BY name

...however it shows all projects to all users. Is there a good way to show only the projects the user has access to?

Modify Ticket

Change Properties


text:
Interesting Determinant

Let $x_1,x_2,\ldots,x_n$ be $n$ real numbers that satisfy $x_1<x_2<\cdots<x_n$. Define \begin{equation*} A=% \begin{bmatrix} 0 & x_{2}-x_{1} & \cdots & x_{n-1}-x_{1} & x_{n}-x_{1} \\ x_{2}-x_{1} & 0 & \cdots & x_{n-1}-x_{2} & x_{n}-x_{2} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ x_{n-1}-x_{1} & x_{n-1}-x_{2} & \cdots & 0 & x_{n}-x_{n-1} \\ x_{n}-x_{1} & x_{n}-x_{2} & \cdots & x_{n}-x_{n-1} & 0% \end{bmatrix}% \end{equation*}

Could you determine the determinant of $A$ in term of $x_1,x_2,\ldots,x_n$?

I make a several Calculation: For $n=2$, we get

\begin{equation*} A=% \begin{bmatrix} 0 & x_{2}-x_{1} \\ x_{2}-x_{1} & 0% \end{bmatrix}% \text{ and}\det (A)=-\left( x_{2}-x_{1}\right) ^{2} \end{equation*}

For $n=3$, we get

\begin{equation*} A=% \begin{bmatrix} 0 & x_{2}-x_{1} & x_{3}-x_{1} \\ x_{2}-x_{1} & 0 & x_{3}-x_{2} \\ x_{3}-x_{1} & x_{3}-x_{2} & 0% \end{bmatrix}% \text{ and}\det (A)=2\left( x_{2}-x_{1}\right) \left( x_{3}-x_{2}\right) \left( x_{3}-x_{1}\right) \end{equation*}

For $n=4,$ we get

\begin{equation*} A=% \begin{bmatrix} 0 & x_{2}-x_{1} & x_{3}-x_{1} & x_{4}-x_{1} \\ x_{2}-x_{1} & 0 & x_{3}-x_{2} & x_{4}-x_{2} \\ x_{3}-x_{1} & x_{3}-x_{2} & 0 & x_{4}-x_{3} \\ x_{4}-x_{1} & x_{4}-x_{2} & x_{4}-x_{3} & 0% \end{bmatrix} \\% \text{ and} \\ \det (A)=-4\left( x_{4}-x_{1}\right) \left( x_{2}-x_{1}\right) \left( x_{3}-x_{2}\right) \left( x_{4}-x_{3}\right) \end{equation*} Finally, I guess that the answer is $\det(A)=2^{n-2}\cdot (x_n-x_1)\cdot (x_2-x_1)\cdots (x_n-x_{n-1})$. But I don't know how to prove it.

-
Your guess would not account for the case $n=2$... –  Arturo Magidin May 14 '12 at 1:56
For $n=2$ is trivial. Maybe for $n>2$ –  tes May 14 '12 at 1:59
I try math induction, but don't know how to connect n by n matrix with (n+1) by (n+1) matrix –  tes May 14 '12 at 2:00
I'll just note that the matrices being considered by the OP, 1. are very similar to Cauchy matrices; 2. can be expressed as the difference of two type D matrices, and 3. have inverses that are cyclic tridiagonal (tridiagonal with extra elements in the upper right and lower left corners). –  Guess who it is. May 14 '12 at 2:45

Clearly the determinant is $0$ if $x_i = x_{i+1}$ (because two adjacent rows are identical) or $x_1 = x_n$ (last row is $-$ first row). So the determinant must be a polynomial divisible by $(x_1 - x_2)(x_2 - x_3) \ldots (x_{n-1} - x_n)(x_n - x_1)$. But the determinant has degree $n$, so it is a constant times this product. To determine what the constant is, you might try a special case: $x_i = i$.

EDIT: Thanks to J.M.'s remark, you can show that in that special case the inverse of your matrix $A_n$ looks like this:

$$\pmatrix{ -\frac{1}{2}+\frac{1}{2n-2} & \frac{1}{2} & 0 & 0 & \ldots & 0 & \frac{1}{2n-2}\cr \frac{1}{2} & -1 & \frac{1}{2} & 0 & \ldots & 0 & 0\cr 0 & \frac{1}{2} & -1 & \frac{1}{2} & \ldots & 0 & 0\cr \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \cr 0 & 0 & 0 & 0 & \ldots & -1 & \frac{1}{2}\cr \frac{1}{2n-2} & 0 & 0 & 0 & \ldots & \frac{1}{2} & -\frac{1}{2} + \frac{1}{2n-2}\cr}$$ where the elements on the main diagonal are all $-1$ except for the first and last, those just above and below the diagonal are all $1/2$, the top right and bottom left are $1/(2n-2)$, and everything else is $0$.

-
Note that $x_i<x_{i+1}$ –  tes May 14 '12 at 2:03
@tes, you didn't use that hypothesis anywhere in your calculations, did you? So the formulas are correct no matter how the variables are related, aren't they? In particular, you're allowed to look at what happens if two are equal, as Robert has done, right? –  Gerry Myerson May 14 '12 at 2:24
I just want to prove that A is invertible –  tes May 14 '12 at 3:12
@robert: How you can conclude that the determinant must be a polynomial divisible by (x1−x2)(x2−x3)…(xn−1−xn)(xn−x1) –  tes May 14 '12 at 3:16
Somewhat apropos: Yueh, in these papers discusses tridiagonal matrices similar to the inverse Robert obtained in this answer. I suspect another proof of the determinant's evaluation can be done based on these (e.g. by expressing the determinant as a product of trigonometric functions of angles in progression). –  Guess who it is. May 14 '12 at 18:04

Expanding Robert solution.

Let $det(A) = P(x)$. Let the polynomial on the right is a multi-variable polynomial $P(x)$.

If $x_1 = x_2$ then $det(A) = 0$ i.e $P(x) = 0$ i.e. $(x_1 - x_2)$ is a factor of $P(x)$.

If $x_2 = x_3$ then $det(A) = 0$ i.e $P(x) = 0$ i.e. $(x_2 - x_3)$ is a factor of $P(x)$.

etc. We calculate possible factors of $P(x)$. Have we calculated all possible factors of $P(x)$?

Let $Q(x) = (x_1 - x_2) (x_2 - x_3) \ldots (x_{n} - x_{1})$

What we know about the degree of $P(x)$? It is $n$, equal to that of $Q(x)$. Thus $Q(x)$ multiplied by some constant factor should give us $P(x)$ i.e. we already have all possible factors of $P(x)$.

A Robert has already mentioned, we should calculate this constant factor.

It follows that if for any $i$, $x_i = x_{i+1}$, then $P(x) =0$ i.e. $det(A) = 0$. Since you alretady have constraints such as $x_1 >x_2 \ldots x_n$, $det(A) \ne 0$.

-
Thank you very much –  tes May 14 '12 at 3:38
Q(x)=(x1−x2)(x2−x3)…(xn−xn−1), where is the x? –  tes May 14 '12 at 3:43
@tes Messed it up. Let me clear it. –  Dilawar May 14 '12 at 3:45
It is not so obvious that the constant is not $0$. –  Robert Israel May 14 '12 at 16:45


text:
## How To Enable the PHP Fileinfo Extension under Windows

Tuesday, 30 November 2010

These five steps will hopefully help you to enable the Fileinfo extension for PHP under Windows. These steps assume you have the extension installed (e.g. php\ext\php_fileinfo.dll). They worked for me using Xampplite 1.7.1 and PHP 5.2.9.

2. Extract the contents of file-4.23-bin.zip\share\file\ to a folder on your machine - I went with path\to\xampplite\php\extras\magic\
3. Add an environment variable, named MAGIC, to your machine with the value pointing at the file named magic - in my example: path\to\xampplite\php\extras\magic\magic
4. Add a line (or uncomment an existing line) to your php.ini: extension=php_fileinfo.dll
5. Restart Apache if it was already running.

That's it!

These steps are probably what you need if you are seeing errors such as these:

• Failed to load magic database
• Fileinfo could not load the magic file
• The given magicfile can not be read
• The given magicfile is not accepted by finfo

(The last two of these error messages are specific to Zend Framework)

References:


text:
# Compute Structure Tensor

As i know, Structure Tensor is:

How i compute $$I_x^2, I_y^2, I_xI_y$$. I used Hadamard product for compute $$I_x^2, I_y^2, I_xI_y$$. But if i use Hadamard product i will get $$det(M) = 0$$. I’m stuck in here for too long. Can anyone explain for me?. Thank you.


text:
# Cohomologies of locally conformally symplectic manifolds and solvmanifolds

created by daniele on 17 Mar 2017
modified on 20 Jul 2017

[BibTeX]

Accepted Paper

Inserted: 17 mar 2017
Last Updated: 20 jul 2017

Journal: Ann. Global Anal. Geom.
Year: 2017
Doi: 10.1007/s10455-017-9568-y
We study the Morse-Novikov cohomology and its almost-symplectic counterpart on manifolds admitting locally conformally symplectic structures. More precisely, we introduce lcs cohomologies and we study elliptic Hodge theory, dualities, Hard Lefschetz Condition. We consider solvmanifolds and Oeljeklaus-Toma manifolds. In particular, we prove that Oeljeklaus-Toma manifolds with precisely one complex place, and under an additional arithmetic condition, satisfy the Mostow property. This holds in particular for the Inoue surface of type $S^0$.