text:
## Friday, November 18, 2005

### Elsevier Science and crackpots

W.S. has pointed out an article written by someone called Ms. Liisa Antilla that has been accepted into a peer-reviewed journal "Global Environmental Change Part A" published by Elsevier Science - the publisher that has done a lot of good things for high-energy physics in the past but whose journal "Nuclear Physics B" is widely considered to be a dinosaur that should die soon, especially because the journal seems much more expensive than what is appropriate in the era of the internet (and the arXiv).

Incidentally, if your university does not provide you with access to the article, you can buy the text for $30. W.S. is amazed by the article, I am also amazed, virtually everyone else is amazed. Why are we so shocked? The title of the article is • Climate of scepticism: US newspaper coverage of the science of climate change and it is arguably the first published peer-reviewed article in the world whose main point is to accuse dozens of particular scientists from scientific misconduct and corruption just because the result of their research does not agree with the author's silly ideological fantasies, without having the tiniest glimpse of evidence. What do I mean? Who is the author? What is it all about? First of all, Lisa Antilla is not a scientist according to all available data. Already the abstract makes you sure that you should not expect the scientific method to play any role in what follows: • This two-part study integrates a quantitative review of one year of US newspaper coverage of climate science with a qualitative, comparative analysis of media-created themes and frames using a social constructivist approach. ... Well, a "social constructivist approach" is one of the postmodern "approaches" whose irrational character - and absurdity - was pointed out by Alan Sokal. The 15 pages make it absolutely clear that Liisa Antilla has no idea about the subject she is writing about - the climate in this case - and her ability to have learned the English alphabet is demonstrably her greatest intellectual achievement. At any rate, this achievement apparently seems sufficient to get published in peer-reviewed journals that belong to Elsevier Science. It must be clear to any rationally thinking person that the chance that an article about "warming" gets published in the newspapers today is at least 5 times higher than the probability that an article that implies "no warming" or "cooling", even locally, gets published - even though cooling and warming are essentially in balance. Bias of one order of magnitude is apparently not high enough to satisfy Liisa Antilla who argues that every article that does not support the crackpot idea of a looming global warming catastrophe must be an artifact of corruption. Her paper has 15 pages and is full of completely chaotic citations of texts about the climate that support the author's unscientific opinions as well as citations of those that disagree with it. The articles that agree with Antilla are assumed to be the holy word while those that disagree with it are creations of the Devil - and the reader is supposed to be equally intellectually challenged. For example, Liisa Antilla cites Naomi Oreskes, another female crackpot whose paper "proving" the "consensus" about the climate change has been shown to be rubbish. The holy side described by Antilla also includes people like Noam Chomsky and random journalists from 255 U.S. newspapers. The "evil" side is represented by scientists such as Christy and Spencer. Antilla proposes theories that all the people who have similar ideas and results about the climate are paid from something like$10,000 dollars that indirectly came from ExxonMobil. I am not sure whether someone's mental capacity can really be so severely compromised so that he or she would believe this kind of stuff - especially when lots of people are getting $1,000,000 grants for doing science of the kind that agrees with the preconceptions of Liisa Antilla and others - but nevertheless this conspiratory theory is good enough for Elsevier Science to be publish it. As mentioned above, Liisa Antilla is not the first female crank of this kind. As a rich person, she has donated money to the Sierra Club (whose description is here) via the Rachel Carson Society (which makes her conspiratory theories about ExxonMobil even more bizarre) and she has signed various anti-industrial petitions. Rachel Carson who gave her name to the society was the very first environmental crackpot. Her book "Silent Spring" (predicting that the birds would be eradicated by DDT) is considered to be completely incorrect today; nevertheless, her flawed ideas - and other, new ideas of similar sort and quality - were apparently good enough for thousands of people to be a part of the so-called "environmental movement". If the political correctness in science continues to thrive, crackpots like Carson, Oreskes, and Antilla - after whom p-branes were named - will completely dominate every individual field in a finite proper time. No one will convince me that there is nothing inherently "female" that these three women share in their approach to reality. It's about a complete inability to figure out how things actually work in Nature combined with a highly exaggerated emphasis on patient reading of all kinds of texts, regardless of their quality, and on making rationally unjustifiable conclusions based on purely verbal patterns of the texts, without any understanding of the content, combined with some irrational prejudices. I don't argue that this approach cannot be found among male researchers; what I argue, however, is that the probability that a female researcher approaches scientific questions in this way is much higher than in the case of males. Do the institutional readers also pay$20,000 per year for a journal that is made of "material" like the article by this particular crackpot? How much does your university pay for such journals?

#### 1 comment:

1. Dear Lumos,

Since you mention Sokal in passing, here's a relevant link on that subject: http://www.mathematik.uni-muenchen.de/~bohmmech/BohmHome/sokalhoax.html

Best wishes,
Nigel


text:
## Calculus: Early Transcendentals (2nd Edition)

$\textbf{T}(t) = \frac{\textbf{r}'(t)}{|\textbf{r}'(t)|}$ Where $\textbf{T}(t)$ is a Unit Tangent Vector and $\textbf{r}'(t)$ is a Tangent Vector on an oriented curve:

format: plain


text:
# How do you simplify (sqrt10 - 9) ^2?

##### 1 Answer

We are going to use the identity

${\left(A - B\right)}^{2} = {A}^{2} - 2 \cdot A \cdot B + {B}^{2}$

Hence we have that

(sqrt10-9)^2=(sqrt10)^2-2*sqrt10*9+9^2= 10-18*sqrt10+81=91-18*sqrt10


text:
# If the volume of box V = 200, L= 12.5, and W = 8, how do you find the height?

Sep 26, 2016

$\text{Height H=} \frac{200}{100} = 2 u n i t .$

#### Explanation:

The Volume $V$ of a Box$= L \cdot W \cdot H$, where, H is the height of the

Box.

Sub.ing the given info. in the above formula, we get,

$200 = \left(12.5\right) \left(8\right) H = 100 H$

$\therefore H = \frac{200}{100} = 2 \text{unit.}$


text:
# rmgpy.molecule.filtration¶

This module contains functions for filtering a list of Molecules representing a single Species, keeping only the representative structures. Relevant for filtration of negligible mesomerism contributing structures.

The rules this module follows are (by order of importance):

1. Minimum overall deviation from the Octet Rule (elaborated for Dectet for sulfur as a third row element)

2. Additional charge separation is only allowed for radicals if it makes a new radical site in the species

3. If a structure must have charge separation, negative charges will be assigned to more electronegative atoms, whereas positive charges will be assigned to less electronegative atoms (charge stabilization)

4. Opposite charges will be as close as possible to one another, and vice versa (charge stabilization)

rmgpy.molecule.filtration.aromaticity_filtration(mol_list, features)

Returns a filtered list of molecules based on heuristics for determining representative aromatic resonance structures.

For monocyclic aromatics, Kekule structures are removed, with the assumption that an equivalent aromatic structure exists. Non-aromatic structures are maintained if they present new radical sites. Instead of explicitly checking the radical sites, we only check for the SDSDSD bond motif since radical delocalization will disrupt that pattern.

For polycyclic aromatics, structures without any benzene bonds are removed. The idea is that radical delocalization into the aromatic pi system is unfavorable because it disrupts aromaticity. Therefore, structures where the radical is delocalized so far into the molecule such that none of the rings are aromatic anymore are not representative. While this isn’t strictly true, it helps reduce the number of representative structures by focusing on the most important ones.

rmgpy.molecule.filtration.charge_filtration(filtered_list, charge_span_list)

Returns a new filtered_list, filtered based on charge_span_list, electronegativity and proximity considerations. If structures with an additional charge layer introduce reactive sites (i.e., radicals or multiple bonds) they will also be considered. For example:

• Both of NO2’s resonance structures will be kept: [O]N=O <=> O=[N+.][O-]

• NCO will only have two resonance structures [N.]=C=O <=> N#C[O.], and will loose the third structure which has the same octet deviation, has a charge separation, but the radical site has already been considered: [N+.]#C[O-]

• CH2NO keeps all three structures, since a new radical site is introduced: [CH2.]N=O <=> C=N[O.] <=> C=[N+.][O-]

• NH2CHO has two structures, one of which is charged since it introduces a multiple bond: NC=O <=> [NH2+]=C[O-]

However, if the species is not a radical, or multiple bonds do not alter, we only keep the structures with the minimal charge span. For example:

• NSH will only keep the N#S form and not [N-]=[SH+]

• The following species will loose two thirds of its resonance structures, which are charged: CS(=O)SC <=> CS(=O)#SC <=> C[S+]([O-]SC <=> CS([O-])=[S+]C <=> C[S+]([O-])#SC <=> C[S+](=O)=[S-]C

• Azide is know to have three resonance structures: [NH-][N+]#N <=> N=[N+]=[N-] <=> [NH+]#[N+][N-2]; here we filter the third one out due to the higher charge span, which does not contribute to reactivity in RMG

rmgpy.molecule.filtration.check_reactive(filtered_list)

Check that there’s at least one reactive structure in the returned list. If not, raise an error (does not return anything)

rmgpy.molecule.filtration.filter_structures(mol_list, mark_unreactive=True, allow_expanded_octet=True, features=None, save_order=False)

We often get too many resonance structures from the combination of all rules, particularly for species containing lone pairs. This function filters them out by minimizing the number of C/N/O/S atoms without a full octet. If save_order is True the atom order is reset after performing atom isomorphism.

A helper function for reactive site discovery in charged species

rmgpy.molecule.filtration.get_charge_span_list(mol_list)

Returns the a list of charge spans for a respective list of :class:Molecule objects This is also calculated in the octet_filtration() function along with the octet filtration process

rmgpy.molecule.filtration.get_octet_deviation(mol, allow_expanded_octet=True)

Returns the octet deviation for a :class:Molecule object if allow_expanded_octet is True (by default), then the function also considers dectet for third row elements (currently sulfur is the only hypervalance third row element in RMG)

rmgpy.molecule.filtration.get_octet_deviation_list(mol_list, allow_expanded_octet=True)

Returns the a list of octet deviations for a respective list of :class:Molecule objects

rmgpy.molecule.filtration.mark_unreactive_structures(filtered_list, mol_list, save_order=False)

Mark selected structures in filtered_list with the Molecule.reactive flag set to False (it is True by default) Changes the filtered_list object, and does not return anything. If save_order is True the atom order is reset after performing atom isomorphism.

rmgpy.molecule.filtration.octet_filtration(mol_list, octet_deviation_list)

Returns a filtered list based on the octet_deviation_list. Also computes and returns a charge_span_list. Filtering using the octet deviation criterion rules out most unrepresentative structures. However, since some charge-strained species are still kept (e.g., [NH]N=S=O <-> [NH+]#[N+][S-][O-]), we also generate during the same loop a charge_span_list to keep track of the charge spans. This is used for further filtering.

rmgpy.molecule.filtration.stabilize_charges_by_electronegativity(mol_list, allow_empty_list=False)

Only keep structures that obey the electronegativity rule. If a structure must have charge separation, negative charges will be assigned to more electronegative atoms, and vice versa. If allow_empty_list is set to False (default), this function will not return an empty list. If it is set to True and all structures in mol_list violate the electronegativity heuristic, the original mol_list is returned (examples: [C-]#[O+], CS, [NH+]#[C-], [OH+]=[N-], [C-][S+]=C violate this heuristic).

rmgpy.molecule.filtration.stabilize_charges_by_proximity(mol_list)

Only keep structures that obey the charge proximity rule. Opposite charges will be as close as possible to one another, and vice versa.


text:
GROUPS:

# Intro

Languages pack an immense ocean of information, but not only as stories they tell in myths, books, songs, anything we see as the memory of civilization. But also languages are a bottomless well of data related to their structure, relationships, origin, and dynamics of their life and death. In celebration of UNESCO International Mother Language Day (FEB 21) let's see what simple notions we can learn about language diversity using pure data and computation. And here is a spark of thought from UNESCO before we start: "To foster sustainable development, learners must have access to education in their mother tongue and in other languages. It is through the mastery of the first language or mother tongue that the basic skills of reading, writing and numeracy are acquired. Local languages, especially minority and indigenous, transmit cultures, values and traditional knowledge, thus playing an important role in promoting sustainable futures."

# Data

Wolfram Language stores numerous built-in data about natural human languages. We can see it in the following properties:

Multicolumn[EntityProperties["Language"], 5]

There are currently in total 10039 languages in the internal database, which keeps being enriched with data:

langs = EntityList["Language"];
langs // Length

10039

but of course many are not, as you can see right below.

# Native speakers

I next compute languages that have number of native speakers listed for them.

popu = DeleteMissing[EntityValue[langs, {"Name", "NativePopulation"}],1, 2];
filt=Select[popu,!StringContainsQ[#[[1]],"script"|"alphabet"]&];
filt//Length

6720

Now I sort languages by native speakers count and show first 100.

barchartPOPU[data_]:=
BarChart[data,
ChartLabels->Automatic,
BarOrigin->Left,
AspectRatio->2,
BaseStyle->14,
ImageSize->{Automatic,1000},
ChartStyle->Opacity[.5],

Row[{
barchartPOPU[Association[Rule@@@SortBy[filt,Last][[-50;;]]]],
barchartPOPU[Association[Rule@@@SortBy[filt,Last][[-100;;-51]]]]
}]

Note, the size of the 1st top bar in the right chart is smaller than the size of the last bottom bar in the left chart - see the axis.

# Geographical distribution

One can also learn about how speakers of a specific language are distributed geographically. Let's define a geo-plotting function

geoPLOT[data_,lbl_]:=
GeoRegionValuePlot[data,
ImageSize->500,
GeoRange -> "World",
GeoProjection->"Robinson",
PlotLabel->Style[lbl,FontFamily->"Phosphate",30,Red],
GeoBackground ->{"CountryBorders", "Land" -> GrayLevel[.9]},
PlotLegends->Placed[Automatic,Bottom]
]

and apply it to the first 8 countries with the largest count of native speakers:

Grid[
Partition[
geoPLOT@@@EntityValue[Interpreter["Language"][Reverse[SortBy[filt,Last][[-8;;,1]]]],{"Speakers","Name"}]
,2],
Frame->All,FrameStyle->GrayLevel[.8]]

# Origin and relations

To me the most fascinating data about languages are origin and relations, such as family or lexical similarity. For example let's consider a quite isolated island society of Philippines.

GeoGraphics[
({EdgeForm[Red], Opacity[0.1], RandomColor[], Polygon[#1]}&)/@ divisions, ImageSize -> 1000,
Epilog -> {Inset[Framed[GeoListPlot[divisions, ImageSize -> 250]], Scaled[{0.85, 0.855}], Automatic],
Inset[Framed[GeoGraphics[GeoMarker[Entity["Country", "Philippines"]],
GeoRange -> "World", GeoProjection -> "Robinson", ImageSize -> 300]],
Scaled[{0.17, 0.93}], Automatic, Scaled[0.4*{1, 1}]]}]

We see above fractured geographic and administrative division structure which perhaps is one of the reasons in past we had formation of complex network of languages. To demonstrate this let's get the data for Lexical Similarity of languages whose Primary Origin is Philippines.

lex=DeleteMissing[EntityValue[langs,{"Name","PrimaryOrigin","LexicalSimilarity"}],1,2];
lex//Length

1092

phil=Cases[lex,{_,Entity["Country", "Philippines"],_}];

I now build a simple network where nodes are languages and an edge exists between any 2 languages having any Lexical Similarity. It is quite fascinating to realize such a complex structure in the same geo-location.

g=Graph[Union[
],
VertexLabels->"Name",GraphStyle->"Prototype",
GraphLayout->"SpringEmbedding",VertexLabelStyle->13,ImageSize->1100]
SetProperty[g,VertexCoordinates->-Reverse/@GraphEmbedding[g]]

# Translation length

Another fun thing to see is typical translation length, which basically tells us do you need more or less characters of a language to express an idea relative to English. We got this info about 180 languages:

trans=DeleteMissing[EntityValue[langs,{"Name","RelativeCharacterCount"}],1,2];
trans//Length

180

We can sort them by translation length and mark in red the top 8 languages with most native speakers shown on the geographic maps above.

Row[{
barchartPOPU[Association[Rule@@@SortBy[trans,Last][[-90;;]]]],
barchartPOPU[Association[Rule@@@SortBy[trans,Last][[-180;;-91]]]]
}]/.(#->Style[#,Red,Bold]&/@SortBy[filt,Last][[-8;;,1]])

Yet another fun way to look at this is using hierarchical clustering with dendrogram. It helps to group languages by capacity of their writing system to deliver the same information per character sequence length. Again we highlight in red languages with the most native speakers. To me it is amazing to see that English clusters by this property with Wayuu and Umbundu rather with its European friends.

This is where I finish my exploration, but look under the long image below for bonus on how to build the top word cloud overlaid over world map. PLease comment if you can come up with more language related computations and explorations.

Dendrogram[Association[
Rule@@@Reverse[SortBy[trans,Last]]],
Right,AspectRatio->3,BaseStyle->12,ImageSize->1000,ClusterDissimilarityFunction->"Average"]/.
(#->Style[#,Red,Bold]&/@SortBy[filt,Last][[-8;;,1]])

# Bonus

This is how you make the top word cloud overlaid with the world map. The words sizes are scaled by native speaking population.

pl1=GeoGraphics[
ImageSize->1000,
AspectRatio->1/2,
GeoRange -> "World",
GeoProjection->"Robinson",
GeoBackground ->{"Coastlines","Land"->Black,"Ocean"->None,"Border"->Black}
];

SeedRandom[2];
pl2=WordCloud[filt,
AspectRatio->1/2,
ImageSize->1000,
WordOrientation->{{-\[Pi]/4,\[Pi]/4}},
WordSpacings->2,
PlotTheme->"Marketing"];

Overlay[{pl1, pl2}]


text:
## Real Analysis Exchange

### Optimal Quantizers for some Absolutely Continuous Probability Measures

Mrinal Kanti Roychowdhury

#### Abstract

The representation of a given quantity with less information is often referred to as ‘quantization’ and it is an important subject in information theory. In this paper, we have considered absolutely continuous probability measures on unit discs, squares, and the real line. For these probability measures the optimal sets of $n$-means and the $n$th quantization errors are calculated for some positive integers $n$.

#### Article information

Source
Real Anal. Exchange, Volume 43, Number 1 (2018), 105-136.

Dates
First available in Project Euclid: 2 May 2018

https://projecteuclid.org/euclid.rae/1525226426

Digital Object Identifier
doi:10.14321/realanalexch.43.1.0105

Mathematical Reviews number (MathSciNet)
MR3816435

Zentralblatt MATH identifier
06924877

#### Citation

Roychowdhury, Mrinal Kanti. Optimal Quantizers for some Absolutely Continuous Probability Measures. Real Anal. Exchange 43 (2018), no. 1, 105--136. doi:10.14321/realanalexch.43.1.0105. https://projecteuclid.org/euclid.rae/1525226426


text:
Taylor K.

asked • 06/20/18

# how many of each does he have?

Ronald has been saving his winning lottery tickets. He has 23 tickets that are worth a total of $175. If each ticket is worth either$5 or \$10, how many of each does he have?

## 1 Expert Answer

By:

Paul M. answered • 06/20/18

BS Mathematics, MD

## Still looking for help? Get the right answer, fast.

Get a free answer to a quick problem.
Most questions answered within 4 hours.

#### OR

Choose an expert and meet online. No packages or subscriptions, pay only for the time you need.


text:
How Many Elements Of The Periodic Table Can You Name? | QuizPug

How Many Elements Of The Periodic Table Can You Name?

Odds are, you haven't even seen a periodic table since high school or college. Which begs the question: how many elements of the periodic table can you name? Are you a scientific genius or will this quiz fall flat? Let's find out!

Question 1/10

pixabay.com
What is the chemical symbol for Hydrogen?
He
H
Fe

Question 2/10

pixabay.com
What is the chemical symbol for Lithium?
Li
Le
Lm

Question 3/10

pixabay.com
What is the chemical symbol for Copper?
Fe
Co
Cu

Question 4/10

pixabay.com
What is the chemical symbol for Potassium?
Pd
K
Pe

Question 5/10

pixabay.com
What is the chemical symbol for Tin?
In
Tn
Sn

Question 6/10

pixabay.com
What is the chemical symbol for silver?
Sm
Se
Ag

Question 7/10

pixabay.com
What is the chemical symbol for Lead?
Ld
Le
Pb

Question 8/10

pixabay.com
What is the chemical symbol for Uranium?
Ur
Um
U

Question 9/10

pixabay.com
What is the chemical symbol for Mercury?
Hg
Na
Mg

Question 10/10

pixabay.com
What is the chemical symbol for Argon?
Ag
Ar
Ae

C+, Chemical Imbalance

pixabay.com
Right!
Okay, so you may know a few basic elements from the periodic table. But this quiz proved that you still have a lot to learn about these chemical elements! Fear not, in taking this quiz, you took a crash course in chemistry and probably learned something new. Go you!

B+, Chemically Clever

pixabay.com
Right!
While many people forgot the periodic table as soon as they learned it back in 8th grade, you maintained a great knowledge of the elements and the symbols that represent them. When we say you're a rarity, we mean it!

A+, Scientifically Savvy

pixabay.com
Right!
Obviously, you excelled in your high school and college chemistry classes! While most struggled to pass this periodic table quiz, you prevailed victorious. It's obvious that you know your elements and their correlating symbols as only a scientifically savvy person can!
Play Again
1
What is the chemical symbol for Hydrogen?
He
H
Fe
2
What is the chemical symbol for Lithium?
Li
Le
Lm
3
What is the chemical symbol for Copper?
Fe
Co
Cu
4
What is the chemical symbol for Potassium?
Pd
K
Pe
5
What is the chemical symbol for Tin?
In
Tn
Sn
6
What is the chemical symbol for silver?
Sm
Se
Ag
7
What is the chemical symbol for Lead?
Ld
Le
Pb
8
What is the chemical symbol for Uranium?
Ur
Um
U
9
What is the chemical symbol for Mercury?
Hg
Na
Mg
10
What is the chemical symbol for Argon?
Ag
Ar
Ae


text:
# One weird tip to improve the success of Data Science projects

Standard

I was recently speaking to some data science friends on Slack, and we were discussing projects and war stories. Something that came across was that ‘data science’ projects aren’t always successful.

Source: pixabay

Somewhere around this discussion a lightbulb went off in my head about some of the problems we have with embarking on data science projects. There’s a certain amount of Cargo cult Data Science and so collectively we as a community – of business people, technologists and executives don’t think deeply enough about the risks and opportunities of projects.

So I had my lightbulb moment and now I share it with everyone.

The one weird trick is to write down risks before embarking on a project.

Here’s some questions you should ask you start a project – preferably gather all data .

• What happens if we don’t do this project? What is the worse case scenario?
• What legal, ethical or reputational risks are there involved if we successfully deliver results with this project?
• What engineering risks are there in the project? Is it possible this could turn into a 2 year engineering project as opposed to a quick win?
• What data risks are there? What kinds of data do we have, and what are we not sure we have? What risks are there in terms of privacy and legal/ ethics?

I’ve found that gathering stakeholders around helps a lot with this, you hear different perspectives and it can help you figure out what the key risks in your project are. I’ve found for instance in the past that ‘lack of data’ killed certain projects. It’s good to clarify that before you spend 3 months on a project.

Try this out and let me know how it works for you! Share your stories with me at myfullname[at]google[dot]com.

# Interviews with Data Scientists: NLP for the win

Standard

Recently I decided to do some quick Data Analysis of my interviews with data scientists.

It seems natural when you collect a lot of data to explore it and do some data analysis on it.

You can access the code here.
The code isn’t in much depth but it is a simple example of how to use NLTK, and a few other libraries in Python to do some quick data analysis of ‘unstructured’ data.

First question:

What does a word cloud of the data look like?

Word cloud of my Corpus based on interviews published on Dataconomy

Here we can see above that science, PHD, science, big etc all pop up a lot – which is not surprising given the subject matter.

Then I leveraged NLTK to do some word frequency analysis. Firstly I removed stop words, and punctuation.

I got the following result – unsurprisingly the most common word was data followed by science, however the other words are of interest – since they indicate what professional data scientists talk about in regards their work.

Source: All interviews published on Dataconomy by me until the end of last week – which was the end of September 2015.

# An interview with a data artisan

Standard

J.D.Long is the current AVP Risk Management at RenaissanceRe and has a 15 year history of working as an analytics professional.

I sent him an interview recently to see what he would say.

1. What project have you worked on do you wish you could go back to, and do better?

Longer answer: Interestingly, what I find myself thinking about when asked this question is not analytics projects where I wish I could redo the analysis, but rather instances where I felt I did good analysis but did a bad job explaining the implications to those who needed the info. Which brings me to #2…

2. What advice do you have to younger analytics professionals?

2) Learn technical skills and enjoy learning new things, naturally. But, 1) always plot your data to visualize relationships and 2) remember at the end of the analysis you have to tell a story. Humans are hard wired to remember stories and not numbers. Throw away your slide deck pages with a table of p values and instead put a picture of someone’s face and tell their story. Or possible show a graph that illustrates the story. But don’t forget to tell the story.

3. What do you wish you knew earlier about being a data artisan?

3) Inside of a firm, cost savings of $1mm seems like it should be the same as generating income of$1mm. It’s not. As an analyst you can kick and whine and gripe about that reality, or you can live with it. One rational reason for the inequality is that income is often more reproducible than cost savings. However, the real reason is psychological. Once a cost savings happens it’s the new expectation. So there’s no ‘credit’ for future years. Income is a little different in that people who can produce \$1mm in income every year are valued every year. That’s one of the reasons I listed “be a profit center” in the post John referenced. There are many more reasons, but that alone is a good one.

4. How do you respond when you hear the phrase ‘big data’?

4) I immediately think, “buzz word alert”. The phrase is almost meaningless. I try to listen to what comes next to see if I’m interested.

5) Everybody loves a good “ah-ha!” moment. Analytics is full of those. I think most of us get a little endorphin drop when we learn or discover something. I’ve always been very open about what I like about my job. I like being surrounded by interesting people, working on interesting problems, and being well compensated. What’s not to love!

Cheers,

P.s. the post J.D.Long mentioned is http://www.johndcook.com/blog/2011/11/21/career-advice-regarding-tools/

# Information Retrieval

Standard

Attention conservation notice: 680 words about Information Retrieval, and highly unoriginal.
The following is very much inspired by a course by Cosma Shalizi but I felt it was worth rewriting to get to grips with the concepts. This is the first of what is hopefully a series of posts on ‘Information Retrieval’, and applications of Mathematics to ‘Data Mining’.
1. Textual features

I’d like to introduce the concepts of how features are extracted, and we shall consider these proxies of actual meanings. One classic representation we can use is bag-of-words (BoW). Let us define this representation, this means we list all of the distinct words in the document together with how often each one appears. This is easy to calculate from the text. Vectors One way we could try to code up the bag-of-words is to use vectors. Let each component of the vector correspond to a different word in the total lexicon of our document collection, in a fixed, standardised order. The value of the component would be the number of times the word appears, possibly including zero. We use this vector bag-of-words representation of documents for two big reasons:

• There is a huge pre-existing technology for vectors: people have worked out, in excruciating detail, how to compare them, compose them, simplify them, etc. Why not exploit that, rather than coming up with stuff from scratch?
• In practice, it’s proved to work pretty well.

We can store data from a corpus in the form of a matrix. Each row corresponds to a distinct case (or instance instance, unit, subject,…) – here, a document – and each column to a distinct feature. Conventionally, the number of cases is n and the number of features is p. It is no coincidence that this is the same format as the data matrix X in linear regression.

2. Measuring Similarity

Right now, we are interested in saying which documents are similar to each other because we want to do a search by content. But measuring similarity – or equivalently measuring dissimilarity or distance</b> – is fundamental to data mining. Most of what we will do will rely on having a sensible way of saying how similar to each other different objects are, or how close they are in some geometric setting. Getting the right measure of closeness will have a huge impact on our results. \paragraph{} This is where representing the data as vectors comes in so handy. We already know a nice way of saying how far apart two vectors are, the ordinary or Euclidean distance, which we can calculate with the Pythagorean formula:

$\displaystyle \|\bar{x} - \bar{y}\| = \sqrt{\sum_{i=1}^{p}(x_i - y_i)^2}$

where ${x_i}$, ${y_i}$ are the ${i^{th}}$ components of ${\bar{x}}$ and ${\bar{y}}$. Remember that for bag-of-words vectors each distinct word – each entry in the lexicon – is a component or a fecture. We can also use our Linear Algebra skills to calculate the Euclidean norm or Euclidean distance . Of any vector this is ${\|\bar{x}\| = \sqrt{\sum_{i=1}^{p}x^{2}_{i}}}$ so the distance between two vectors is the norm of their distance ${\bar{x} - \bar{y}}$. Equivalently, the norm of a vector is the distance from it to the origin, ${\bar{0}}$ Obviously, one can just look up a topology textbook and remind oneself of other metrics such as the taxicab metric.

2.1. Normalisation

Just looking at the Euclidean distances between document vectors doesn’t work, at least if the documents are at all different in size. Instead, we need to normalise by document size, so that we can fairly compare short texts with long ones. There are (at least) two ways of doing this.
Document length normalisation Divide the word counts by the total number of words in the document. In symbols,

$\displaystyle \bar{x} \mapsto \frac{\bar{x}}{\sum_{i=1}^{p} x_{i}}\ \ \ \ \ (1)$

Notice that all the entries in the normalised vector are non-negative fractions, which sum to 1. The i-th component is thus the probability that if we pick a word out of the bag at random, it’s the i-th entry in the lexicon.

Cosine ‘distance’ is actually a similarity measure, not a distance`:

$\displaystyle d_{cos} \bar{x}, \bar{y} = \frac{\sum_{i} x_{i}y_{i}}{\|\bar{x}\|\|\bar{y}\|} \ \ \ \ \ (2)$

It’s the cosine of the angle between the vectors ${\bar{x}}$ and ${\bar{y}}$.


text:
Deep RL Course documentation

A Q-Learning example

Join the Hugging Face community

to get started

# A Q-Learning example

To better understand Q-Learning, let’s take a simple example:

• You’re a mouse in this tiny maze. You always start at the same starting point.
• The goal is to eat the big pile of cheese at the bottom right-hand corner and avoid the poison. After all, who doesn’t like cheese?
• The episode ends if we eat the poison, eat the big pile of cheese or if we spent more than five steps.
• The learning rate is 0.1
• The gamma (discount rate) is 0.99

The reward function goes like this:

• +0: Going to a state with no cheese in it.
• +1: Going to a state with a small cheese in it.
• +10: Going to the state with the big pile of cheese.
• -10: Going to the state with the poison and thus die.
• +0 If we spend more than five steps.

To train our agent to have an optimal policy (so a policy that goes right, right, down), we will use the Q-Learning algorithm.

## Step 1: We initialize the Q-table

So, for now, our Q-table is useless; we need to train our Q-function using the Q-Learning algorithm.

Let’s do it for 2 training timesteps:

Training timestep 1:

## Step 2: Choose action using Epsilon Greedy Strategy

Because epsilon is big = 1.0, I take a random action, in this case, I go right.

## Step 3: Perform action At, gets Rt+1 and St+1

By going right, I’ve got a small cheese, so $R_{t+1} = 1$, and I’m in a new state.

## Step 4: Update Q(St, At)

We can now update $Q(S_t, A_t)$ using our formula.

Training timestep 2:

## Step 2: Choose action using Epsilon Greedy Strategy

I take a random action again, since epsilon is big 0.99 (since we decay it a little bit because as the training progress, we want less and less exploration).

I took action down. Not a good action since it leads me to the poison.

## Step 3: Perform action At, gets Rt+1 and St+1

Because I go to the poison state, I get $R_{t+1} = -10$, and I die.

## Step 4: Update Q(St, At)

Because we’re dead, we start a new episode. But what we see here is that with two explorations steps, my agent became smarter.

As we continue exploring and exploiting the environment and updating Q-values using TD target, Q-table will give us better and better approximations. And thus, at the end of the training, we’ll get an estimate of the optimal Q-function.


text:
# (100-100) Divided By (100-100) Is Equal To 2. Prove It.

$frac{100 – 100}{100 – 100} = 2$

= $frac{100 – 100}{100 – 100} = frac{(10)^{2} – (10)^{2}}{10 (10 – 10)}$

= $frac{(10 + 10)(10 – 10)}{10 (10 – 10)}$

= $frac{(10 + 10)}{10}$

= $frac{20}{10}$

= 2

$frac{100 – 100}{100 – 100} = 2$

Hence proved

Explore more such questions and answers at BYJU’S.


text:
Least Squares Fitting - 68% Confidence Interval

I am fitting a linear polynomial to some data and I have derived the errors for each of the best-fit parameters from the covariance matrix. I would expect these errors to correspond to a $1\sigma$ 68% confidence interval, but I am finding this is not the case.

If I instead do a grid search, where I freeze one of the two parameters and grid the other, searching for the parameter values corresponding to $\chi^{2}_{\mathrm{min}} + 1$, I get a smaller error interval than from the least squares errors. According to Bevington (Data Reduction and Error Analysis for the Physical Sciences), a single-parameter 68% confidence interval is given by parameter values that increase the $\chi^{2}$-value from $\chi^{2}_{\mathrm{min}}$ to $\chi^{2}_{\mathrm{min}}+1$.

I have tested this with multiple codes and they all give the same results. Can someone help me to understand this behavior?

• Better on Cross Validated? – dmckee May 24 '16 at 20:24
• Is your covariance matrix diagonal? If it's not, then your two parameters are correlated. Your second analysis (grid search) seems to be implicitly assuming that your two parameters are uncorrelated. – Paul T. May 25 '16 at 12:30
• @dmckee yes, I think this is off topic here – David Z May 25 '16 at 12:36

I think what you have described is not the correct method to estimate a 68% confidence interval in one parameter of interest.

The error is in freezing the other parameter when minimising chi-squared.

A better procedure to evaluate the uncertainty in parameter 1, is to evaluate the minimum chi-squared for a set of values of parameter 1, whilst allowing parameter 2 to vary. The best fit parameter 1 value is at the global minimum, whilst an estimate of its uncertainty is provided by where the chi-squared increases by 1 from this value, but not necessarily at the same value of parameter 2.

If you freeze parameter 2 at its best fit value, you will underestimate the uncertainty in parameter 1. The reasons for this is that the locus of least chi-squared fit in a parameter 1 vs parameter 2 space is (in general) inclined with respect to these axes.

The best way to do it is to evaluate chi-squared over all the parameter space and then find the projection of the chi-squared+1 contour onto each of the parameter axes. A picture may say a thousand words. The red arows show how you tried to do it. The blue limits show my (economical) first way of trying to improve the estimate and then the black arrows show the projection of the error ellipse onto the x-axis.

• In other words, for 68% in 1D find the (possibly discontiguous ) region in $x$ where$\min_{y,z\cdots}\chi^2(x, y, z,\cdots) - \min_{x,y,z\cdots}\chi^2(x, y, z,\cdots) ≤ 1$ – innisfree May 25 '16 at 13:38
• Thanks! This makes sense to me. It seems the error from least squares correctly gives you the 1$\sigma$ projection of the error ellipse on the parameter axis for a line when the ellipse is inclined as a result of correlations between parameter 1 and parameter 2. It looks like you can also get the errors from projecting the error ellipse onto the parameter axes using the covariance matrix. How would you derive 1 sigma confidence intervals if you have more than 2 parameters (say, 5 parameters) and several of them are correlated with each other? – grover May 27 '16 at 4:20
• @grover The expensive way is to calculate an entire grid and project along each axis. Monte Carlo might be cheaper. Or you increment one param, minimise chisq wrt the rest and use this 1d plot to find where chisq increases by +1. This is computationally cheap and roughly correct. Note that it is only chisq+1 if you are interested in only one parameter. – Rob Jeffries May 27 '16 at 6:06

In the case of a 2-parameter fit, the 68% confidence space is typically an ellipse, not necessarily aligned with the axes. If you want to figure out the size of the ellipse, you should find the orientation of the ellipse, not just the intercepts of the ellipse with a horizontal and vertical line through its center.

Example: fit $y=a+b x$ for a large set of data points clustered around $(x,y)=(1,1)$. The quality of the fit will be pretty similar for $(a,b)=(1,0)$ or $(0,1)$. But if you change $a$ without changing $b$ in the opposite direction (this is what you did), you will very quickly get a bad fit. But this doesn't tell you much about the uncertainty in $a$.

• The procedure I use is to iteratively change one parameter without changing the other. Then, I search for $\chi^{2}_{\mathrm{min}}$ and update the value. This process is repeated for each parameter until $\chi^{2}_{\mathrm{min}}$ does not change and you obtain a global minimum. This guarantees you are getting the best fit by doing an exhaustive search. Then, you can derive confidence intervals from looking at parameter values corresponding to $\chi^{2}_{\mathrm{min}} + 1$. My question is why is this different from the errors one derives from the least squares covariance matrix? – grover May 24 '16 at 20:21
• That what the answer tells you: because the error ellipse is not guaranteed to be axis-aligned. – dmckee May 24 '16 at 20:53


text:
# Depletion layers in undoped semiconductors

I've been studying the effects of surface states in semiconductors (in the absence of interfaces, so just semiconductor-vacuum), and am struggling with the concept of depletion/accumulation layers. I've read quite a few texts on the subject, and each of them always deals with n-type or p-type semiconductors. But I don't know what would happen in the case of the undoped semiconductor.

Take for example the case of an n-type semiconductor with acceptor surface states as depicted in the figure below, not so graciously taken from Solid Surfaces, Interfaces and Thin Films by Lüth. One has acceptor states around the middle of bandgap that are partially filled, resulting in a negative surface charge. This negative charge is then compensated by the depletion layer, consisting of ionized donors.

Conversely, one could take partially filled donor like states near the bandgap middle and a p-type semiconductor; this results in a positively charged surface and a negative depletion layer due to occupied bulk acceptors.

But now my question comes: how do these scenario's work for undoped semiconductors? Surface charges will still form, and they still need to be compensated by space charge layers, but in the absence of donor/acceptor dopants to be depleted/occupied, how does this happen?

• If the surface states are really mid-gap, and the Fermi level in the intrinsic material is mid-gap, what is the problem? The Fermi levels line up just fine without requiring any formation of depletion layers to get the levels to line up. – Jon Custer Mar 24 '17 at 15:21
• @JonCuster Okay, that's true, and a flaw in my question. What if they are not truly midgap? Take InSB, where the surface states are close to the valence band (just inside the gap). Or InAs, where they are above the conduction band. – user129412 Mar 24 '17 at 15:25
• So that will pin the Fermi level at the surface. Now what? Well, just because there isn't doping doesn't mean there aren't carriers around to move. – Jon Custer Mar 24 '17 at 15:30
• Okay, but that becomes a bit tricky right? I mean, I can see how a normal lattice atom (instead of a dopant) in the depletion region might lose one of its electrons, but the opposite already sounds stranger; it would have to host an extra one. Or well, perhaps that is not the right way to look at it. I could see an accumulation layer form in the case of InAs, where more electrons than usual are present around the surface, causing a net negative charge layer, and perhaps in the case of a depletion layer it just means that less electrons are there. – user129412 Mar 24 '17 at 15:37
• But still this becomes quite tricky, as all the expressions in textbooks (schottkey barriers and such) always use $N_D$, the donor doping concentration, for the thickness of such layers. What determines the thickness in undoped systems then? – user129412 Mar 24 '17 at 15:38


text:
# how to convert sentence that contains “no more than 3” into predicate logic sentence?

How to convert sentence that contains “no more than 3” into predicate logic sentence?

For example: "No more than three $$x$$ satisfy $$R(x)$$" using predicate logic.

This is what I have for "exactly one $$x$$ satisfies $$R(x)$$": $$\exists x(R(x) \land \forall y(R(y) \rightarrow (x = y)))$$

$$\forall x \forall y \forall z \forall u ((R(x)\wedge R(y) \wedge R(z) \wedge R(u)) \rightarrow (x=y \vee x=z \vee x=u \vee y=z \vee y=u \vee z=u))$$
$$\exists x \exists y \exists z \forall u (R(u) \rightarrow (u = x \lor u = y \lor u = z))$$


text:
# matplotlib.axes.Axes.set_ylim¶

Axes.set_ylim(bottom=None, top=None, emit=True, auto=False, **kw)

Set the data limits for the y-axis

Parameters: bottom : scalar, optional The bottom ylim (default: None, which leaves the bottom limit unchanged). top : scalar, optional The top ylim (default: None, which leaves the top limit unchanged). emit : bool, optional Whether to notify observers of limit change (default: True). auto : bool or None, optional Whether to turn on autoscaling of the y-axis. True turns on, False turns off (default action), None leaves unchanged. ylimits : tuple, optional The bottom and top yxlims may be passed as the tuple (bottom, top) as the first positional argument (or as the bottom keyword argument). ylimits : tuple Returns the new y-axis limits as (bottom, top).

Notes

The bottom value may be greater than the top value, in which case the y-axis values will decrease from bottom to top.

Examples

>>> set_ylim(bottom, top)
>>> set_ylim((bottom, top))
>>> bottom, top = set_ylim(bottom, top)

One limit may be left unchanged.

>>> set_ylim(top=top_lim)

Limits may be passed in reverse order to flip the direction of the y-axis. For example, suppose y represents depth of the ocean in m. The y-axis limits might be set like the following so 5000 m depth is at the bottom of the plot and the surface, 0 m, is at the top.

>>> set_ylim(5000, 0)


text:
What is some significant symbolism (relevance) in The Book Thief?

I'm just stuck on this one question, and it's due tomorrow. Thanks for the help!!


text:
## Student Algebraic Geometry Seminar: Geometry of the moduli space of curves of genus $g$

Seminar | April 17 | 4-5 p.m. | 891 Evans Hall

Ritvik Ramkumar, UC Berkeley

Department of Mathematics

Given a variety $X$ it's a basic question to ask if it's rational i.e. admits a birational map $P^n \to X$ for some $n$. For curves and surfaces there are explicit criterion that determine when a variety is rational. Answering this question in higher dimensions is much more difficult. In light of this, it's easier to ask for a weaker question: Does there exist a dominant rational map $P^n\to X$. In this case, $X$ is said to be unirational. Over the complex numbers, every unirational curve or surface is rational. For curves this is a consequence of the Riemann-Hurwitz Formula.

A very important geometric object is $M_g$, the moduli space of curves of genus $g$. We can ask for what $g$, if any, is $M_g$ rational or unirational. In 1915, Severi proved that $M_g$ is unirational for $g\leq 10$ and conjectured that it's unirational for all $g$. However, in 1987, it was shown that $M_g$ is of general type if $g\geq 24$. In this talk I will review classical examples of unirational varieties, outline a modern proof of Severi's result and describe related conjectures and results. I will finish by describing how the geometry of $M_g$ changes when $g\geq 11$.

events@math.berkeley.edu


text:
# 编译原理（五）

## 考虑以下文法，写出每个非终端符号的 FIRST 集和 FOLLOW 集

$S\to aTUV\vert bV\\ T\to U\vert UU\\ U\to\varepsilon\vert bV\\ V\to\varepsilon\vert cV$
FIRST FOLLOW
$S$ $a,b$ $\text{\textdollar}$
$T$ $\varepsilon,b$ $\text{\textdollar},b,c$
$U$ $\varepsilon,b$ $\text{\textdollar},b,c$
$V$ $\varepsilon,c$ $\text{\textdollar},b,c$

## 考虑以下文法

$S\to(L)\vert a\\ L\to L,S\vert S$

### 消除文法的左递归

$S\to(L)\vert a\\ L\to ST\\ T\to ,ST\vert\varepsilon$

### 构造文法的 LL(1) 分析表

FIRST FOLLOW
S $(\ a$ $\text{\textdollar}\ )\ a$
L $(\ a$ $)$
T $,\ \varepsilon$ $)$

$($ $)$ $a$ $,$ $\text{\textdollar}$
S $S\to(L)$   $S\to a$
L $L\to ST$   $L\to ST$
T   $T\to\varepsilon$   $T\to,ST$

### 对于句子 $(a, (a, a))$，给出语法分析的详细过程（参照课本 228 页的图 4.21）

MATCH STACK IN OUT
$S\text{\textdollar}$ $(a,(a,a))\text{\textdollar}$
$(L)\text{\textdollar}$ $(a,(a,a))\text{\textdollar}$ $S\to(L)$
$($ $L)\text{\textdollar}$ $a,(a,a))\text{\textdollar}$
$($ $ST)\text{\textdollar}$ $a,(a,a))\text{\textdollar}$ $L\to ST$
$($ $aT)\text{\textdollar}$ $a,(a,a))\text{\textdollar}$ $S\to a$
$(a$ $T)\text{\textdollar}$ $,(a,a))\text{\textdollar}$
$(a$ $,ST)\text{\textdollar}$ $,(a,a))\text{\textdollar}$ $T\to,ST$
$(a,$ $ST)\text{\textdollar}$ $(a,a))\text{\textdollar}$
$(a,$ $(L)T)\text{\textdollar}$ $(a,a))\text{\textdollar}$ $S\to L$
$(a,($ $L)T)\text{\textdollar}$ $a,a))\text{\textdollar}$
$(a,($ $ST)T)\text{\textdollar}$ $a,a))\text{\textdollar}$ $L\to ST$
$(a,($ $aT)T)\text{\textdollar}$ $a,a))\text{\textdollar}$ $S\to a$
$(a,(a$ $T)T)\text{\textdollar}$ $,a))\text{\textdollar}$
$(a,(a$ $,ST)T)\text{\textdollar}$ $,a))\text{\textdollar}$ $T\to,ST$
$(a,(a,$ $ST)T)\text{\textdollar}$ $a))\text{\textdollar}$
$(a,(a,$ $aT)T)\text{\textdollar}$ $a))\text{\textdollar}$ $S\to a$
$(a,(a,a$ $T)T)\text{\textdollar}$ $))\text{\textdollar}$
$(a,(a,a$ $)T)\text{\textdollar}$ $))\text{\textdollar}$ $T\to\varepsilon$
$(a,(a,a)$ $T)\text{\textdollar}$ $)\text{\textdollar}$
$(a,(a,a)$ $)\text{\textdollar}$ $)\text{\textdollar}$ $T\to\varepsilon$
$(a,(a,a))$ $\text{\textdollar}$ $\text{\textdollar}$

## 考虑以下文法，这一文法是否是 LL(1) 文法？给出理由

$S\to aSbS\vert bSaS\vert\varepsilon$


text:
# The core -symmetric players

We have $n$-persons ($n\ge 3$) cooperative game. And we know that player $1$ and $2$ are symmetric. So for each element $(x_1,x_2,...,x_n)$ from the core we have $x_1=x_2$ ? Is that true ? Never seen example when this is not true, but how do it come from ?

Consider the following three person TU-game in the usual lexi-order of coalitions

v=[0,0,0,8,5,5,14]

Notice, that in this game players $1$ and $2$ are substitutes, i.e., the transposition (1,2) is a symmetry in the game. Computing the core-vertices of the game, we get

      0   8   6
0   9   5
9   5   0
5   9   0
8   0   6
9   0   5

From these vertices, we observe that it does not hold for all core elements $x_{1}=x_{2}$.


text:
# Michael Bell

Ph.D. Students Blog

## Annotation maturity: Comparison of annotations in new and old sets of UniProtKB entries.

3 February, 2012 (14:57) | Uncategorized | By: mj_bell

Carrying on from the previous post, we now wish to look at annotation maturity in sets of UniProtKB entries. We have seen that over time the quality of annotations appear to be decreasing over time, for both Swiss-Prot and TrEMBL. A reasonable explanation for this would be that annotations are constantly being added to the newly incorporated data, which in turn has added additional pressure on curators, meaning that over time the least effort has shifted from the reader to the annotator. Whilst we see a reduction in the overall annotation quality, we suspect that a mature set of entries would improve over age.

To approach this analysis, we compare annotations from entries that are in both Swiss-Prot Version 9 and UniProtKB/Swiss-Prot version 15. We also show the resulting alpha value for annotations from entries in UniProtKB/Swiss-Prot version 15 but not Swiss-Prot Version 9. The resulting graph for this is shown below:

With the assumption that maturity is linked to age, we would expect that the quality of annotations within a set of old entries would improve over time. Interestingly, this doesn’t appear to be the case. Whilst the alpha value does decrease (by roughly 0.1), the alpha value for the remaining entries is significantly lower. This would suggest that annotations within the whole database are generally decreasing, although the rate of decrease depends on initial age of the entry. Given this, it is of interest to see how the quality of annotations in only new entries change over time. For this, we extract annotations for those entries that appeared for the first time in a given version. The resulting graph for this is shown below:

This graph also shows a steady decrease over time – a similar pattern to most of our previous analyses. These results are interesting; it would appear that annotation of new data is getting worse over time. It also appears to have a detrimental affect on other annotations. We have discussed the increase in data in relation to annotation quality, but the impact of size alone would not explain the decrease of annotation quality in older entries. One possible explanation for this is the protocol used for annotation curation.

The curation of annotations is clear process, consisting of 6 key steps. This process is detailed in , with an overview of the process shown in the figure below (taken from ).

Part of the protocol is to, for a given sequence, identify similar entries and then standardise and propagate annotation between these entries to ensure data consistency. Presumably, over time the curation process has undergone revisions (worthy of further investigation and another blog post!) due to changes in resources, increase of data, and so on. It is possible that this curation process is refined to deal with larger amounts of data and quicker release dates (both of which are true for Swiss-Prot over time – early versions of Swiss-Prot saw around 1000 new entries being added, with the later versions seeing around 30,000 new entries, whilst the release cycle is more frequent by a couple of months). Although the increase of manually curated entries and faster release dates could be due to more curators rather than change in annotation protocol (which will be investigated further), it is plausible that attempts to standardise annotation between similar entries is actually having a detrimental affect on overall annotation quality.

## Annotation maturity: Average entry age within UniProtKB

26 January, 2012 (11:04) | Uncategorized | By: mj_bell

As we know the UniProtKB database is growing exponentially. Our prior quality analysis of whole UniProtKB versions has seen that, over time, annotations are becoming more geared towards the annotator; that is, readers now require more effort to interpret and digest the annotations than in earlier versions of UniProtKB. The exponential growth of the database gives a plausible explanation to this – the initial quality of annotations cannot be maintained. This explanation also fits with other research .

Given this, we would like to abstract away from the growth of the database and see how sets of annotations change over time. For example, how do annotations for a set of entries in both Swiss-Prot Version 9 and Swiss-Prot version 45 compare? Our initial step is to investigate the average age of entries within UniProtKB.

Within each entry there is a set of date stamps, as explained in the UniProtKB manual, one of which indicates when the entry first appeared in the database. For each entry within a database version we extract this date and calculate the average. The figure below shows the average entry age for both TrEMBL and Swiss-Prot:

This graph shows that the average entry age increases over time. These results aren’t unexpected, as new data is constantly being added, which will gradually push the average age up. Although entries in Swiss-Prot come from TrEMBL, they overwrite the initial TrEMBL date and state the date integrated into Swiss-Prot, hence why we see a similar pattern for both Swiss-Prot and TrEMBL. Additionally, older entries can become deleted or merged. Very little information is given about what happens to the dates for merged entries. It isn’t uncommon to see two or more entries becoming merged; when this happens, one accession number is used as a primary accession number with the remaining accession numbers becoming secondary accession numbers. The primary accession isn’t necessarily the oldest entry and we only extract dates from primary accession numbers.

Whilst this graph shows that the average entry age is increasing, the difference between the release date and average entry date also shows an initial increase. We currently have over 20 years worth of Swiss-Prot versions, with the average record age for the latest version being around 5 years old, compared to version 9, which is around a year and a half. This difference between the average and and release date is shown below:

To illustrate this, the first graph shows that Swiss-Prot version 9 was released in November 1988 and that the average entry release date was July 1987 – the difference between these dates is 1 year and 4 months, which is reflected in the bottom left point in the above graph. This graph is slightly different than what we previously expected. Initially we see the age difference increasing, but then start to decrease (Swiss-Prot) or level off (TrEMBL); we would expect a constant increase. We assume age and maturity are linked; something old is more mature than something young. If this is indeed the case, then this would suggest Swiss-Prot is decreasing in maturity, whilst TrEMBL remains at a steady level of maturity. It could be suggested that the increase in entries hit a maximum level, before having a detrimental effect. A similar graph and suggestion could be shown by, for example, the response time for a server with a constant increase in concurrent users querying a database.

Although viewing entry dates, rather than annotation quality, this conclusion fits with that of our other analyses; the rate of data is outstripping our ability to deal with it. Following on from this, we now wish to look at maturity of annotations within entry sets. This is discussed in the next blog post (link)

## Most frequently occurring sentences in UniProtKB and their propagation through the Web

4 November, 2011 (16:16) | Sentence reuse, Uncategorized | By: mj_bell

As already established, sentence reuse is common within UniProtKB. Obviously, some sentences will have higher reuse than others, giving an indication as to their information content. Below we show the top 10 sentences for the first versions of Swiss-Prot and TrEMBL, and also for Version 15 of UniProtKB/TrEMBL and UniProtKB/Swiss-Prot.

Top 10 sentences for Swiss-Prot Version 9:

 Sentence Occurrences dimer of identical chains. 141 tetramer of identical chains. 125 to other ef-hand calcium binding proteins. 94 ferredoxin are iron-sulfur proteins that transfer electrons in a wide variety of metabolic reactions. 79 these proteins may be involved in dormant spore’s high resistance to uv light. 73 nadh + ubiquinone = nad(+) + ubiquinol. 69 4 ferrocytochrome c + o(2) = 2 h(2)o + 4 ferricytochrome c. 68 cytoplasmic. 63 avian ovomucoid consist of three homologous tandem kazal family inhibitory domains. 63 subunit i, ii, and iii form the functional core of the enzyme complex. 52

Top 10 sentences for UniProtKB/Swiss-Prot Version 15:

 Sentence Occurrences cytoplasm (by similarity). 69247 homodimer (by similarity). 28182 cytoplasm. 26648 nucleus (by similarity). 17013 cytoplasm (potential). 16029 secreted. 15020 nucleus. 14855 monomer (by similarity). 13922 cytoplasm (probable). 12340 the rnap catalytic core consists of 2 alpha, 1 beta, 1 beta’ and 1 omega subunit. 11460

Top 10 sentences for TrEMBL Version 1:

 Sentence Occurrences d-ribulose 1,5-bisphosphate + co(2) = 2 3-phospho-d-glycerate. 538 integral membrane protein. 488 belongs to the cytochrome p450 family. 460 4 ferrocytochrome c + o(2) = 2 h(2)o + 4 ferricytochrome c. 374 subunit i and ii form the functional core of the enzyme complex. 374 nuclear. 370 subunit ii binds cu(a) and cytochrome c. 370 electrons originating in cytochrome c are transferred via heme a and cu(a) to the binuclear center formed by heme a3 and cu(b). 362 copper a and heme group. 362 to other mitochondrial or bacterial cox2 subunits. 362

Top 10 sentences for UniProtKB/TrEMBL Version 15:

 Sentence Occurrences the sequence shown here is derived from an embl/genbank/ddbj whole genome shotgun (wgs) entry which is preliminary data. 1518744 endonucleolytic cleavage to 5′- phosphomonoester. 262057 mitochondrion inner membrane; multi-pass membrane protein (by similarity). 96557 contains 1 reverse transcriptase domain. 84828 cytochrome c oxidase is the component of the respiratory chain that catalyzes the reduction of oxygen to water. 83892 subunits 1- 3 form the functional core of the enzyme complex. 83892 cytoplasm (by similarity). 81184 contains 1 peptidase a2 domain. 71851 belongs to the heme-copper respiratory oxidase family. 64848 4 ferrocytochrome c + o(2) + 4 h(+) = 4 ferricytochrome c + 2 h(2)o. 63881

It is interesting to view which sentences come out top, and how these change over time. One of the most obvious things to note is the drastic reduction in sentence length for Swiss-Prot over time, whereas TrEMBL sustains a similar sentence length.

However, whilst this view is interesting, the main purpose is as an initial test to see how sentences from UniProtKB propagate through the web, if at all. We would expect annotations to flow to various other places, whether it is blogs and webpages such as this, or to more formal places, such as other databases. Checking for the flow of the most popular sentences will give an indication as to whether further investigation for lesser used sentences is worthwhile.

As an initial and simple test we can search for exact sentences which contain five or more words on Google. We choose sentences with 5 or more words as it is less likely a matching sentence would be independently curated by chance. Additionally, sentences with 5 words or more are seen to hold more information content (this will be discussed more in future blog posts).

An example result can be viewed for the sentence “cytochrome c oxidase is the component of the respiratory chain that catalyzes the reduction of oxygen to water.”, which occurs with high frequency in UniProtKB/TrEMBL version 15. It is hard to get a reasonable figure as to how many Google hits these results return (Results are, at best, a rough estimate. The way these estimations are calculated cause particular errors for some of the sentences we are searching for, given we only want exact sentences. More information about this can be found here and here.) but we can see the sentence has spread into a number of other places. These include STRING, PhosphoSite, DrugBank and a paper published in the Neurobiology of Aging journal.

Some of these pages state that the information was pulled from UniProtKB, with some others linking to UniProtKB, whilst others do not make any reference to UniProtKB. We do have to consider that UniProtKB could have pulled this sentence from another location or the sentence was curated elsewhere by chance, albeit unlikely for all cases. Given the sentence appeared in a journal article it is possible this is the source of the sentence, which coincides with the UniProtKB curation protocol.

Being unable to easily tell where the original source of this data isn’t uncommon and highlights one of the major problems we are attempting to address. We know some of the sites pull the data from UniProtKB, and assume those mentioning UniProtKB also do, even if not explicitly stated. This can cause major issues should this annotation be found to have been erroneous. Do these sites do regular checks, either automated or manual, for changes to the original annotation/entry, or would this annotation remain unchanged? This simple example highlights the difficulty with detecting and tracing error propagation and provenance, and its subsequent impact on assessing an annotations quality and correctness.

## UniProtKB and Benford’s Law

5 October, 2011 (14:36) | Annotation Quality, Miscellaneous | By: mj_bell

In the last blog post we looked at data parsing, and how Zipf’s law could possibly be used to detect parsing errors. Whilst reading a recent blog post by Ben Goldacre I was reminded of Benford’s law – which shares a number of similarities to Zipf’s – and considered how it may also be applicable to detecting parsing errors.

Like Zipf’s Law, Benford’s Law is a rather interesting empirical law. The law states that the occurrences of first digits for a set of numbers aren’t evenly distributed. By first (or leading) digit, we simply mean that we take only the first digit of a number, regardless of its size. So the leading digit of 18362 is 1, whilst the leading digit of 489 is 4. Interestingly, the chance of the first digit being 1 is around 30%, whereas the chance of the leading digit being 9 is around 5% – like the distribution of grid-line widths on the logarithmic scale. For a given set of numbers, if, for each leading digit $d (d \in {1, ..., 9})$ occurs with probability $P(d) = log_{10}(d+1) - log_{10}(d) = log_{10} (1 + \frac{1}{d})$ then the numbers are said to satisfy Benford’s law.

Also like Zipf’s Law, Benford’s Law has been seen to hold in numerous applications, assuming the data is distributed over multiple orders of magnitude. However, one of the most interesting applications of Benford’s law is the detection of fraud. When numbers are being fabricated, in an attempt to disguise fraud, people appear to distribute the numbers evenly, when naturally this isn’t the case. Given this, we are interested to see if UniProt also follows Benford’s Law and if it can be used for detecting parsing errors.

We have created a number of graphs for Swiss-Prot and TrEMBL for Benford’s law. For each graph we have plotted orange diamonds to show the distribution predicted by Benford’s law. On each graph the X-axis represents leading digits with the corresponding percentage being shown on the Y axis. Below are the graphs showing Swiss-Prot version 9 and UniProtKB/Swiss-Prot Version 15:

These graphs show that Swiss-Prot follows a Benford distribution. This was also true for intermediate versions of Swiss-Prot. Additionally, we did the same for TrEMBL, with graphs for versions 1, 28 and UniProtKB/TrEMBL version 15 shown below:

For TrEMBL the results are much more scattered, with later versions not really following Benford’s Law. We appear to be able to extract similar meaning from these graphs as we did from the Zipf ones. For example, it shows Swiss-Prot is more “natural” than TrEMBL and that early versions of TrEMBL are of better quality than later versions.

Whilst this view gives some further confidence to the underlying annotation, investigation into error detection isn’t as clear. One of the main issues is we don’t have much erroneously parsed data to check – only those that contain copyright and topic header “errors”. Using Benford’s Law we were unable to detect any “errors” in these datasets. Data that produced a major skew for Zipf’s law produced negligible impact on Benford’s law. However, it is likely that different kinds of parsing errors could be detected with Benford’s law, that would be unnoticeable by Zipf’s law. A combination of both methods could be employed when checking parsed data, and any subsequent work on this would be interesting to see.

## Have I parsed my data correctly?

22 September, 2011 (16:06) | Miscellaneous | By: mj_bell

The foundations for most of our work has been based on data parsed from text files. Extracted data has included single words and whole sentences from gigabytes of raw text. With such overwhelming amounts of data, how can we be confident that we have correctly parsed our data?

Obviously some basic checking was performed. This included counting the number of entries expected vs number parsed, manual checking of random entries that have been parsed, parsing artificially created data and so on. Doing checks such as these help detect parsing errors; in our case we identified a mismatch of expected entries parsed (which was an error with the UniProt release notes, which they have subsequently corrected). However, as with all testing, you can only show the presence of bugs/errors, not prove that parsed data is error free. In many cases you also don’t actually know everything you need to test for.

As detailed in previous blog posts, we have been applying power laws to our parsed data. In a number of cases the resulting graphs have given unexpected results – such as artifactual kinks or outlying alpha values. Upon inspection these were due to incorrectly parsed data. In the first instance, a kink in the tail of the graph was due to incorrectly parsing copyright statements. These statements are an example of an error we couldn’t foresee or test for. Similarly, we also detected the incorrect parsing of topic block headers. Another error was due to incorrect escaping of speech marks when reading data dumps (OK, not a parsing error, but still an error detected with this method).

It appears that a side-effect of our original analysis has been the detection of incorrectly parsed data. The usage of Zipf’s Law is common; numerous papers exist that claim a Zipfian distribution has been found in all kinds of natural and man-made phenomena. This is also true of similar empirical laws, such as Pareto’s law. Given this, it isn’t unreasonable to hypothesise that power-law approaches could be applied to the detection of incorrectly parsed data.

I am not aiming to investigate this further, rather I found it an interesting side-effect of our analysis. Should this approach be explored, it would be necessary for the algorithm to distinguish between parsing errors and ‘real’ outliers. It would also have to be available in a way that meant it was easy to use and quick to run (in proportion to the total parsing time). It would be interesting to see if such an approach could reliably detect errors in parsed data, and given the generic nature of these approaches, the amount of literature on the subject and my experience I would suspect so.

## Anaylsing selenocysteine sentence reuse and flow.

31 August, 2011 (11:17) | Annotation Quality, Sentence reuse, Website | By: mj_bell

The sentence “Selenocysteine is encoded by the opal codon.” was previously discussed, having been identified through a network view of sentence usage within Swiss-Prot version 9. Our interest was mainly due to it being present in two entries in different interconnected groups. In that blog entry we looked at the similarities (or lack of) between the entries that shared the sentence. Given the development of dynamic graphs (as discussed in the last blog post) we thought it worth revisiting to see what additional information we can extract from the annotation space. Firstly, we can look at the graph for sentence occurrences over time:

This sees a steady increase of the sentence reuse in Swiss-Prot, until Version 44 when it steeply drops from 81 occurrences to 9. We can also see that in TrEMBL there is only ever one occurrence. Previously we knew that the sentence was last seen in UniProtKB/Swiss-Prot version 8, when it was only in 2 occurrences, however we didn’t know about the big decline in its usage before its removal, nor could we easily tell how many entries where made up from TrEMBL and Swiss-Prot. The removal of this sentence appears to be due to the incorporation of selenocysteine information into an entries feature table. We are unsure when Selenocysteine information was added to the feature table; presumably it was after Swiss-Prot Version 44, with the remaining entries initially being missed. There doesn’t appear to be any other clear reason for this sharp decline.

We can also view which entries the sentence occurs in, and for which database versions:

In this view, each point represents a sentence that was in a given entry (as stated on the x-axis) for a given version (release date on the y-axis). A red point indicates the entry is in the TrEMBL dataset, whilst blue indicates inclusion in Swiss-Prot. This view shows a couple of interesting points:

1. Reappearance within an entry
2. In two entries (P18283 and P12079) it is interesting to see the sentence being removed and then later reappearing. In both cases, the sentence is removed after Swiss-Prot Version 23, with it reappearing in version 38 for P18283 and version 42 for P12079. Looking at the history of P18283 and P12079 we can see that the sentence was “replaced” with “THE ACTIVE-SITE SELENOCYSTEINE IS ENCODED BY THE OPAL CODON, UGA (BY SIMILARITY).”. The usage of “by similarity” is mainly used in TrEMBL, where the information is inferred computationally. This is illustrated below, with the corresponding graph for the similarity sentence:

Whilst the sentence is in the majority of TrEMBL entries, it does appear in some Swiss-Prot entries.

Reverting back to the original sentence, in the latest version of P12079 (which has merged with P11352) a comment has been added that states “sequence was originally thought to originate from human.” Looking at the history of both entries this confusion appears to have led to the uncertainty about the selenocysteine annotation. The sentence reappears in P12079 when it is merged with P11352. There is no clear indication in P18283 why the sentence was reinstated. In the latest version of both these entries, the encoding of selenocysteine is documented in the feature table.

3. Transient appearance
4. In one entry (P21765) we see the sentence appearing, but in the subsequent version it is removed. It only appears in a single version of the entry. In the latest version of the entry there is no mention of selenocysteine, so it would appear this was an erroneous annotation. The entry Q49613 also only appears a single time in second graph, however the entry was deleted after one version.

5. Multiple entry merges
6. A large majority of the entries that share the sentence in the graph are merged in later versions. For example, in the latest version accessions P22352; O43787; Q86W78; Q9NZ74 and Q9UEL1 are all merged into a single entry. In our graph, they are all independent entries containing the sentence. This is also the case for a number of other entries on the graph, including the final two entries to have the sentence.

The analysis above gave some interesting cases and patterns — but what value can we take from this? Whilst this is only a single example, we can start to see possible uses and applications for this data. However, all of these will require further analysis:

• Annotation error rate in Swiss-Prot has been estimated to be as high as 43% . We have seen that annotations that are transient or reappeared appear to have an annotation error or entry error. This could suggest that more stable annotations are more likely to be correct (paper link).
• As discussed and explored during the network analysis blog post, this approach allow us to see entries for which a sentence is (or was) included. We have seen that it is possible for entries with limited sequence similarity to share common annotations. This approach allows UniProt entries to be browsed for shared commonality through annotation, as opposed to just sequence similarity.
• The first graph showed levels of sentence reuse within Swiss-Prot and TrEMBL, culminating with a large decline, before complete removal. In this instance the sharp decline signified the inclusion of the sentence into the feature table, but major changes or fluctuations for other sentences could indicate additional information, such as a large erroneous annotation propagation.
• Levels of reuse can give an indication to the quality of the sentence. A heavily reused sentence is typically more generic than a sentence that is unique to a single entry. This suggests unique sentences are more ‘meaningful’ than generic sentences, as quantified using approaches such as Inverse Document Frequency and Term Frequency-Inverse Document Frequency .
• As stated in the curation protocol , annotations are standardised across homologous proteins. This means it is possible that the inclusion of a sentence in an entry will propagate to homologous entries for a given database version. This is shown in the graph, and is expected. Should the sentence be removed from one of these entries, it is also possible that the sentence will be removed from the homologous entries. We can use these approaches to determine if the sentence still exists amongst any of these entries.

Each sentence has to originate in an individual entry (or group of entries in the same version). In this particular example, the sentence originated in two entries, and it was still in these entries before removal. Should the sentences have been removed from the original entries, it may be worth reassessing other entries that subsequently include this sentence. If the sentence was found to be false in the root entry, then it possible that an error has propagated into numerous other entries.

• We have seen in this example that the majority of entries sharing this sentence have been merged. It is possible that an entry is merged with another entry, or entries, which do not share a particular sentence. We could hypothesise that a merged entry which shared a common sentence prior to a merge means the sentence is more likely to be correct than one which isn’t shared by the other entry (or entries).

As a final point, it is worth mentioning sentence similarity. We have purposefully focused on the reuse of exact sentences, as we know they are standardised and copied between entries as part of the curation protocol , thus allowing us to analyse annotation flow and reuse. However, we can see that there are sentences that are very similar in the database:

• the active-site selenocysteine is encoded by the opal codon, uga.
• the active-site selenocysteine is encoded by the opal codon, uga (by similarity).
• the selenocysteine is encoded by the opal codon, uga.
• the active- site selenocysteine is encoded by the opal codon, uga.
• the active-site selenocysteine is encoded by a normal cys codon.

In one case the difference is an extra space (“active- site”), which is only in a single entry and likely a copy and paste typo. For the sentence “the selenocysteine is encoded by the opal codon, uga.”, it is shown in entries P26970 and P26971, with it later being replaced with “the active-site selenocysteine is encoded by the opal codon, uga.” in Swiss-Prot version 30. These first 4 sentences are sementically similar; that is the message they convey is pretty much the same, and you could argue they should all be treated as the same sentence.

However we need to be careful with some sentences that appear highly similar. For example, the last sentence (“the active-site selenocysteine is encoded by a normal cys codon.”) states that selenocysteine is encoded by a cysteine codon, not the opal codon, in the entries it is used in. Whilst it is semantically similar in the sense selenocysteine is encoded by a particular codon, it is done so with a different codon. Therefore, investigating sentences that are semantically similar appears to be worthwhile, but care must be taken if treating similar sentences “as one”.

## Producing web-based dynamic graphs

5 August, 2011 (13:05) | Sentence reuse, Uncategorized, Website | By: mj_bell

As part of my work on sentence reuse I have been investigating ways to visualise various sets of data on my website. An obvious requirement of this is that the graphs must be developed dynamically; the resulting graph depending on a users query. We also have to account for various types of data, not just numerical data. For example, we want to produce a graph that shows, for each database version, which entry (or entries) a given sentence occurs in. Below is an early mock-up of such a graph in R, illustrating this example:

From this graph we can see that for the given sentence, it initially only occurs in two database entries, before propagating through the database. We can also see in some instances where it has been removed from an entry and then reappearing in a later version. This current view has a number of issues, with perhaps the main issue being it isn’t always clear which point corresponds to a database entry and accession number. The mock-up also doesn’t make it clear which entries come from TrEMBL and which are Swiss-Prot, whilst the labels should also be more meaningful (release dates on the Y axis and actual accession numbers on X axis).

A range of tools looking to meet these requirements were tried and tested, including:

A number of issues were identified with most approaches. These included:

• The majority of approaches made use of the canvas element, which isn’t supported by Internet Explorer (without using additional scripts)
• Reliance on a third party server (couldn’t run all code locally)
• No ability to handle non-numeric data types
• Problems with licencing

We opted to use the HighCharts project for our charts, as it appeared to meet our full requirements. Benefits of HighCharts are that it is open source, provides numerous chart types, ability to zoom sections of a graph, ability to display tooltip text and the ability to export and save graphs. As an example, we have produced a graph similar to the one above, this time dynamically created in HighCharts:

As can be seen, it overcomes some of the drawbacks of the previous approach. TrEMBL entries are clearly visible in red, whilst Swiss-Prot entries are in blue. The axis labels are also more meaningful and we can hover over any point to get a tool-tip, clearly identifying which accession number and version it relates to. Data points are also click-able, providing links to further information, whilst you can also zoom into areas of the graph.

The production of these graphs is part of an overall view we are producing to allow information about sentence reuse in UniProtKB to be analysed and browsed. We will blog about further developments in future posts.

## Levels of sentence reuse in UniProtKB

26 July, 2011 (12:35) | Sentence reuse, Uncategorized | By: mj_bell

As is frequently highlighted, data being added to biological databases is ever increasing; typically at an exponential rate. This is true for the number of entries added over time to both UniProtKB/Swiss-Prot and UniProtKB/TrEMBL, as illustrated below:

UniProtKB offer a number of detailed statistics for each release of Swiss-Prot and TrEMBL, including the total number of species represented (including which species are most represented), sequence sizes, total number of references, and so on. These statistics are used for various analyses, such as analysing database growth and total coverage. As discussed previously we have started to analyse sentence reuse within UniProtKB, and can provide additional statistics from a sentence viewpoint. For example, we can see that the total number of sentences within Swiss-Prot and TrEMBL are also increasing at an exponential rate:

This level of growth isn’t unexpected. As more entries are added to UniProtKB, then more sentences are either added or copied to these entries, as each entry contains at least a single sentence of annotation. It is interesting to see that TrEMBL and Swiss-Prot had similar numbers of sentences until the last four versions, where TrEMBL convincingly overtook Swiss-Prot. Perhaps a more interesting perspective is to look at how many unique sentences occur within each dataset:

This shows a steady linear increase of unique sentences within Swiss-Prot. For TrEMBL the level of uniqueness appears to maintain a similar level, but a clearer indication is shown below:

The levels, whilst initially show a constant increase, fluctuate erratically. Finally, we can view the percentage of unique sentences in both TrEMBl and Swiss-Prot:

This shows that the level of unique sentences decrease over time. It shows a steady decline for Swiss-Prot, whilst TrEMBL seems to jump down in steps. Given that TrEMBL is created automatically, these jumps and the erratic levels of unique sentences perhaps relate to changes in the annotation algorithm. One of our reasons for analysing sentence reuse is to determine which sentences have more information content than others.

The more frequently a sentence occurs, the more generic it is. Therefore, a sentence that is unique to an individual entry would be expected to have a better information content than one that appears in numerous entries. For example, the sentences “atp + h(2)o + h(+)(in) = adp + phosphate + h(+)(out).”, “contains 1 ring-type zinc finger” and “monomer (by similarity).” all occur in numerous entries whereas the sentences “has weak antifungal activity toward c.comatus and p.piricola”, “the ligand for sev is the boss (bride of sevenless) protein on the surface of the neighboring r8 cell.” and “brain; in the cal region of hippocampus, the medial habenula, and raphe nuclei.” only occur in one entry within a dataset. Given this, and the levels of reuse illustrated earlier, this would suggest that sentences that are unique to an entry, or show very little levels of reuse, provide better quality information to a reader.

## Selenocysteine is encoded by the opal codon.

6 May, 2011 (13:17) | Networks, Sentence reuse | By: mj_bell

As mentioned previously, sentence reuse is common in both manual and automated annotation curation. Although our analysis is in its early stages, we have already noticed that, in most cases, sentences are only shared between clusters of proteins. In Swiss-Prot version 9 we noticed one sentence, “THE ACTIVE-SITE SELENOCYSTEINE IS ENCODED BY THE OPAL CODON, UGA.”, which was only linked between two proteins (P07203 and P07658) that were in separate clusters. To illustrate this, the image below shows an example of a sentence linked between two clusters. In the image each node is a protein entry and each edge is a sentence that appears in both its connected nodes.

Due to the rarity of linked groups this sentence ‘jumped out’. Whilst this was enough to warrant further investigation, the fact that selenocysteine was apparently encoded by a stop codon (UGA) within these two proteins made it even more interesting. Upon investigation I found out that opal codons can do more than just terminate translation (8132075), and that the co-translational incorporation of selenocysteine is confirmed in a number of papers (e.g. (8744353)).

It is still interesting to see no clear link between the two entries other than this single sentence. One is a protein from a prokaryotic organism (Escherichia coli) and another from Eukaryotic organism (Homo sapien). A BLAST search showed very little sequence similarity between the two entries (even using the various versions of sequences). Additionally the stated functions of the two proteins are very different. These differences would suggest that a manual curator using the approach given by UniProt would not consider these two proteins at the same time and they would be curated independently.

A possible explanation regarding the reuse of the sentences would be that they both reference the same paper. However the supporting papers ( and for P07203 and (2941757) for P07658) are different. None of the papers actually state the sentence as shown in the annotation, share any of the same authors or reference each other. This would suggest the sentence was manually written and copied between the two entries manually.

In later versions of the database we see the sentence occur in a total of 84 entries, and was last seen in UniProtKB/Swiss-Prot Version 8 (P83564) and UniProtKB/TrEMBL Version 24 (P83564). The sentence has, presumably, been removed due to the mutation being added to the each entries feature table. Being able to browse annotations in this way is currently cumbersome and labour intenstive so we are currently developing tools to aid this and these tools will be discussed in the blog as they near (beta) completion.

Whilst our analysis is in the very early stages, we can already show a link between two proteins that would been seen as unrelated in the traditional sense via functional annotation.

## UniProtKB and Sentence Reuse

25 April, 2011 (12:26) | Annotation Quality, Sentence reuse | By: mj_bell

Previously we have looked at the occurrences of words within UniProtKB and seen varying degrees of reuse. What about whole sentences? When an annotation is created, either manually or automatically, annotations are frequently reused (i.e. sentences copied and pasted from one entry to another), as detailed in the curation protocol . By investigating this reuse, we can look at tracking annotation flow through database versions, analyse sentence distribution and provide a useful interface to this data by offering, for example, a concordance view.

Having extracted sentences for use in this analysis, it required little additional effort to extract the occurrence of each sentence for each database version. By doing this we thought it would be interesting to initially see if fitting a power law to the sentences would give similar results to those from fitting to words. This was done for both Swiss-Prot and TrEMBL. Graphs for UniProtKB/TrEMBL version 1 and 13 are shown below:

The TrEMBL graphs show the same pattern as the word graphs; over time, the amount of reuse increases heavily. Below shows graphs for Swiss-Prot version 20 and UniProtKB/Swiss-Prot version 11:

Like TrEMBL, these graphs also follow the pattern of the word graphs. Over time it is clear Swiss-Prot becomes more mature with the clear development of of two slopes.

These graphs show that sentence reuse occurs in both TrEMBL and Swiss-Prot and the body of reused sentences also increases. Given this information, it would appear highly likely that analysing the sentence distribution would be worthwhile.


text:
A dataset consisting of 1000 observations drawn from a 14-component normal mixture in which the covariances of the components have the same size and shape but differ in orientation.

data(wreath)

## References

C. Fraley, A. E. Raftery and R. Wehrens (2005). Incremental model-based clustering for large datasets with small clusters. Journal of Computational and Graphical Statistics 14:1:18.


text:
# Velocity obstacle

The velocity obstacle VOAB for a robot A, with position xA, induced by another robot B, with position xB and velocity vB.

In robotics and motion planning, a velocity obstacle, commonly abbreviated VO, is the set of all velocities of a robot that will result in a collision with another robot at some moment in time, assuming that the other robot maintains its current velocity.[1] If the robot chooses a velocity inside the velocity obstacle then the two robots will eventually collide, if it chooses a velocity outside the velocity obstacle, such a collision is guaranteed not to occur.[1]

This algorithm for robot collision avoidance has been repeatedly rediscovered and published under different names: in 1989 as a maneuvering-board approach,[2] in 1993 it was first introduced as the "velocity obstacle",[3] in 1998 as collision cones,[4] and in 2009 as forbidden velocity maps.[5] The same algorithm has been used in maritime port navigation since at least 1903.[6]

The velocity obstacle for a robot ${\displaystyle A}$ induced by a robot ${\displaystyle B}$ may be formally written as

${\displaystyle VO_{A|B}=\{\mathbf {v} \,|\,\exists t>0:(\mathbf {v} -\mathbf {v} _{B})t\in D(\mathbf {x} _{B}-\mathbf {x} _{A},r_{A}+r_{B})\}}$

Variations include common velocity obstacles (CVO),[7] finite-time-interval velocity obstacles (FVO),[8] generalized velocity obstacles (GVO),[9] hybrid reciprocal velocity obstacles (HRVO),[10] nonlinear velocity obstacles (NLVO),[11] reciprocal velocity obstacles (RVO),[12] and recursive probabilistic velocity obstacles (PVO).[13]

## References

1. {{#invoke:Citation/CS1|citation |CitationClass=journal }}
2. {{#invoke:citation/CS1|citation |CitationClass=conference }}
3. {{#invoke:citation/CS1|citation |CitationClass=conference }}
4. {{#invoke:Citation/CS1|citation |CitationClass=journal }}
5. {{#invoke:citation/CS1|citation |CitationClass=conference }}
6. {{#invoke:citation/CS1|citation |CitationClass=book }}
7. {{#invoke:citation/CS1|citation |CitationClass=conference }}
8. {{#invoke:citation/CS1|citation |CitationClass=conference }}
9. {{#invoke:citation/CS1|citation |CitationClass=conference }}
10. {{#invoke:citation/CS1|citation |CitationClass=conference }}
11. {{#invoke:citation/CS1|citation |CitationClass=conference }}
12. {{#invoke:citation/CS1|citation |CitationClass=conference }}
13. {{#invoke:citation/CS1|citation |CitationClass=conference }}