text:
#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Options

# Saving climate data

edited December 2016

Climate scientists are watching with increasing alarm the inquiries and movements of the incoming Trump administration not only with respect to pursuit and funding of climate science, but also ejecting climate scientists who have contributed to it at NOAA and NASA and, presumably, related government-funding organizations. Worse, there is a concern that the climate observing assets so carefully built up over two decades will be shut down, and possibly destroyed, with data loss. See

Accordingly meteorologist Eric Holthaus has begun an effort to download, document, and stash in safe places all climate related data stored on .gov computers. There's a lot of it, and I do not know details about this effort, especially pertaining to how they will be sure the metadata are preserved. I have reached out to ask.

Given the importance and prominence of these data to work done at Azimuth, I thought the membership would be interested in knowing about this, if not helping.

I would suggest communications about it be kept as much off social media as possible.

Thanks. I'll let people know when I learn anything more.

• Options
1.
edited December 2016

We're happy to help, but it's a big job and it would go faster if it were put on social media. In any event, Eric Holthaus is already tweeting about it. For example this tweet of his was shared 495 times two hours after he posted it:

Scientists: Do you have a US .gov climate database that you don't want to see disappear? Add it here: https://docs.google.com/spreadsheets/d/12-__RqTqQxuxHNOln3H5ciVztsDMJcZ2SVs1BrfqYCc/edit#gid=0 Please share

Let us know what we can do.

Comment Source:We're happy to help, but it's a big job and it would go faster if it were put _on_ social media. In any event, Eric Holthaus is already tweeting about it. For example [this tweet](https://twitter.com/EricHolthaus/status/807802102416228352) of his was shared 495 times two hours after he posted it: > Scientists: Do you have a US .gov climate database that you don't want to see disappear? Add it here: [https://docs.google.com/spreadsheets/d/12-__RqTqQxuxHNOln3H5ciVztsDMJcZ2SVs1BrfqYCc/edit#gid=0](https://docs.google.com/spreadsheets/d/12-__RqTqQxuxHNOln3H5ciVztsDMJcZ2SVs1BrfqYCc/edit#gid=0) Please share Let us know what we can do.
• Options
2.
edited December 2016

We're happy to help, but it's a big job and it would go faster if it were put on social media.

I am not sure wether this "stashing in safe places all climate related data stored on .gov computers." is a good idea. At least it should be made dammed clear that the amount of information which is lost by such a "rushed transfer" is huge.

Like if I think of the methane measurements all the info about concrete measurements procedures etc. would not be included. In the end it may in some cases mean that the corresponding data is completely useless without the missing context info (which is by the way bad in itself with regard to future uses -but thats another topic).

I think those "climate deniers" you are talking about and Trump himself understand that for example data like a "Storm events database" is not only important for climate science but also for air and space travel, likewise "Sea level trends" for nautical purposes etc. It is harder to explain that even "paleoclimatology datasets" might be useful for this purposes, but you know it can be done in principle. They should know that if they destroy the corresponding infrastructure than this is combined with a total or at least huge loss of this data. I can't imagine that they would like to take responsibility for that unless they are in bad autodestruct mode - which unfortunately happens not too seldom with leaders if they are on the losing side (see e.g. the example of germans sending their own youth (or here) into a lost battle) , but this is not the case here at least not right now.

Comment Source:>We're happy to help, but it's a big job and it would go faster if it were put on social media. I am not sure wether this "stashing in safe places all climate related data stored on .gov computers." is a good idea. At least it should be made dammed clear that the amount of information which is lost by such a "rushed transfer" is huge. Like if I think of the methane measurements all the info about concrete measurements procedures etc. would not be included. In the end it may in some cases mean that the corresponding data is completely useless without the missing context info (which is by the way bad in itself with regard to future uses -but thats another topic). I think those "climate deniers" you are talking about and Trump himself understand that for example data like a "Storm events database" is not only important for climate science but also for air and space travel, likewise "Sea level trends" for nautical purposes etc. It is harder to explain that even "paleoclimatology datasets" might be useful for this purposes, but you know it can be done in principle. They should know that if they destroy the corresponding infrastructure than this is combined with a total or at least huge loss of this data. I can't imagine that they would like to take responsibility for that unless they are in bad autodestruct mode - which unfortunately happens not too seldom with leaders if they are on the losing side (see e.g. the example of germans sending their own youth (or here) into a lost battle) , but this is not the case here at least not right now.
• Options
3.
edited December 2016

I agree that it might be overreacting to assume Trump will want to destroy all repositories of climate data. However, it's hard to estimate the probability that he'll do this, so it makes some sense to back it up as a precaution. It would actually be good to do this regardless of what happens - events other than the election of Trump could someday destroy this data.

Of course it's also good for people to try to prevent Trump from trying to destroy the government repositories. These are independent activities.

Apparently some people at Google and other big internet companies are helping Holthaus. I need to find out if that's true. If they're doing a good enough job, I can focus on adding databases to the list.

Everything suggests that Trump and his cabinet may want to completely stop climate research. First, the guy managing his EPA (Enviornmental Protection Agency) transition team is one of the world's most effective climate change deniers, and under George W. Bush he had tried to get the head of the EPA fired:

Second, Trump's planned head of the EPA, Scott Pruitt, led 28 states in a lawsuit against the EPA to block Obama's Clean Power Plan:

Third, his favored candidate for Secretary of State, Rex Tillerson, was the head of Exxon and won a contract for exploring Arctic waters for oil:

Fourth and fifth:

This last story is what's getting people really scared.

So, I'll keep adding databases to the list.

Comment Source:I agree that it might be overreacting to assume Trump will want to destroy all repositories of climate data. However, it's hard to estimate the probability that he'll do this, so it makes some sense to back it up as a precaution. It would actually be good to do this _regardless_ of what happens - events other than the election of Trump could someday destroy this data. Of course it's _also_ good for people to try to prevent Trump from trying to destroy the government repositories. These are independent activities. Apparently some people at Google and other big internet companies are helping Holthaus. I need to find out if that's true. If they're doing a good enough job, I can focus on adding databases to the list. Everything suggests that Trump and his cabinet may want to completely stop climate research. First, the guy managing his EPA (Enviornmental Protection Agency) transition team is one of the world's most effective climate change deniers, and under George W. Bush he had tried to get the head of the EPA fired: * John Baez, [This man must be stopped](https://plus.google.com/u/0/117663015413546257905/posts/1tKE8kik9sx), 10 November 2016. Second, Trump's planned head of the EPA, Scott Pruitt, led 28 states in a lawsuit against the EPA to block Obama's Clean Power Plan: * Matt McGrath, [Trump nominee to rekindle climate battle?](http://www.bbc.com/news/science-environment-38249208), _BBC_, 9 December 2016. Third, his favored candidate for Secretary of State, Rex Tillerson, was the head of Exxon and won a contract for exploring Arctic waters for oil: * Philip Bump, [Who is Rex Tillerson, the ExxonMobil chairman who may become secretary of state?](https://www.washingtonpost.com/news/the-fix/wp/2016/12/10/who-is-rex-tillerson-the-exxonmobil-chairman-who-may-become-secretary-of-state/), _Washington Post_, 10 December 2016. Fourth and fifth: * Jason Samesnow, [Trump adviser proposes dismantling NASA climate research](https://www.washingtonpost.com/news/capital-weather-gang/wp/2016/11/23/trump-adviser-proposes-dismantling-nasa-climate-research/?tid=a_inl), _Washington Post_, 23 November 2016. * Christopher Dean Hopkins, [Trump transition asks Energy Department which employees work on climate change](http://www.npr.org/sections/thetwo-way/2016/12/09/505041927/trump-transition-asks-energy-dept-which-employees-work-on-climate-change), _National Public Radio_, 9 December 2016. This last story is what's getting people really scared. So, I'll keep adding databases to the list.
• Options
4.
edited December 2016

so it makes some sense to back it up as a precaution

My whole point above was that I think that this is unfortunately not a "backup" and that some people might be lured into thinking that this is a "backup". It's not.

Comment Source:>so it makes some sense to back it up as a precaution My whole point above was that I think that this is unfortunately not a "backup" and that some people might be lured into thinking that this is a "backup". It's not.
• Options
5.

Well, presumably if the Web sites themselves are mirrored, any and all files documenting methodology, calibration, collection techniques, and interpretations would be saved as well. If there are paper-only records which are at risk, not much we can do to help about them. Moreover, a truly paranoid person might want to provide an alternate control plane for the observational platforms in question, with encrypted controls, given as a safeguard to scientists in another government, with a way of locking out the existing controls.

Comment Source:Well, presumably if the Web sites themselves are mirrored, any and all files documenting methodology, calibration, collection techniques, and interpretations would be saved as well. If there are paper-only records which are at risk, not much we can do to help about them. Moreover, a truly paranoid person might want to provide an alternate control plane for the observational platforms in question, with encrypted controls, given as a safeguard to scientists in another government, with a way of locking out the existing controls.
• Options
6.

Do people mirror entire ftp sites like those on the list to be saved? Of course I assume the US government has them backed up, but we're talking about backing them up in some other way.

So far I haven't seen any evidence that anyone is actually doing anything about this problem, other than listing the databases that need to be saved. I emailed a bunch of people about this, and I'll email a bunch more, but if you learn anything helpful let me know! I don't myself have the technical ability to efficiently copy an entire database.

Comment Source:Do people mirror entire ftp sites like those [on the list to be saved](https://docs.google.com/spreadsheets/d/12-__RqTqQxuxHNOln3H5ciVztsDMJcZ2SVs1BrfqYCc/edit#gid=0)? Of course I assume the US government has them backed up, but we're talking about backing them up in some _other_ way. So far I haven't seen any evidence that anyone is actually doing anything about this problem, other than listing the databases that need to be saved. I emailed a bunch of people about this, and I'll email a bunch more, but if you learn anything helpful let me know! I don't myself have the technical ability to efficiently copy an entire database.
• Options
7.
edited December 2016

John, so I have just started trying to mirror all of CDIAC. We'll see. I'll put it in a tarball, and then throw it up on Google. It should keep everything intact. Using WinHTTrack. I have coordinated with Eric via Twitter, creating, per your suggestion, a new personal account which I am using exclusively to follow the principals.

Once CDIAC is done, and checked over, I'll move on to other sites.

There are things beyond our control, such as paper records, or records which are online but are not within visibility of the public.

Oh, and I've formally requested time off from work for latter half of December so I can work this on vacation. (I have a number of other projects I want to work in parallel, anyway.)

Comment Source:John, so I have just started trying to mirror all of CDIAC. We'll see. I'll put it in a tarball, and then throw it up on Google. It should keep everything intact. Using WinHTTrack. I have coordinated with Eric via Twitter, creating, per your suggestion, a new personal account which I am using exclusively to follow the principals. Once CDIAC is done, and checked over, I'll move on to other sites. There are things beyond our control, such as paper records, or records which are online but are not within visibility of the public. Oh, and I've formally requested time off from work for latter half of December so I can work this on vacation. (I have a number of other projects I want to work in parallel, anyway.)
• Options
8.
edited December 2016

Great, Jan! That's really good news! Anything I can do to help? Any resources you need?

Comment Source:**Great, Jan!** That's really good news! Anything I can do to help? Any resources you need?
• Options
9.

Do people mirror entire ftp sites

I never worked with HTTrack, but it doesn't look as if it mirrors ftp contents: https://en.wikipedia.org/wiki/HTTrack Does it?

But even if -a lot of ftp sites are only accessible per (non-public) login. And then again you miss out specifications written on paper and a lot of other knowledge in particular the one which is in the brain of the scientists.

But apart from that "download technicalities" it is moreover crucial that curves are updated, like this Carbon dioxide curve:

In short, Jan what you do is somewhat "taking fotographs" of the climate science webcontent. This is of course better than nothing, but it doesn't replace the "living thing". So again - I think it is important to make clear that THIS IS NO BACKUP and one shouldn't lure politics into believing that it is!

By the way by looking at the CO2 curve I asked myself how one could express the whole problemacy of CO2 emissions a bit more directly.

From the curve I picked three points (by handwaving read off estimation). That is in 1960 one had a CO2 microfraction of 310, 25 years later 340 and 50 years later in 2010 one had a microfraction of 410. Or in direct air fractions 0.00031, 0.00034 and 0.0004 that is 0.031, 0.034 and 0.041 per cent. If I make the ansatz that this is some curve obeying the function $$C+B e^{A*T}$$ and set 1960 to be start time (I.e. T=0) I get: $$287.5+22.5e^{0.0339 T}$$ as an approximating curve. For this you don't need big maths, just e-function and quadratic equations, i.e. undergraduate college math.

According to Wikipedia Donald Trump graduated from Penn in May 1968 with a Bachelor of Science degree in economics, so this kind of math should have been part of his education i.e. eventually with a little fresh up he could in principle perform the calculations himself.

So with this approximation one reaches a value of 70000 microfractions, i.e. 0.07, i.e. 7 percent air fraction at which you get already mental confusion due to CO2 poisoning within 237 years- if I haven't messed up somewhere in the calculations (please check!). Or in other words:

if CO2 emissions go on like this then even people hiding out on mountains to avoid CO2 laden cities will be poisoned within only 237 years.

Again this is a calculation where you dont need big models of climate science, you just need to trust the measurements or use your own homemade measurements (which will be analogous).

And yes this is only an ansatz but even if this would not be 237 years but 1000 years or worse in the other direction within the lifetime of our kids- it is clear that mankind leaves a carbon footprint which might have the sooner or later even direct health effects. Maybe one should communicate these directly to understand facts, rather then say "climate scientists calculated."

Comment Source:>Do people mirror entire ftp sites I never worked with HTTrack, but it doesn't look as if it mirrors ftp contents: https://en.wikipedia.org/wiki/HTTrack Does it? But even if -a lot of ftp sites are only accessible per (non-public) login. And then again you miss out specifications written on paper and a lot of other knowledge in particular the one which is in the brain of the scientists. But apart from that "download technicalities" it is moreover crucial that curves are updated, like this Carbon dioxide curve: ![carbondioxide](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Mauna_Loa_CO2_monthly_mean_concentration.svg/708px-Mauna_Loa_CO2_monthly_mean_concentration.svg.png) In short, Jan what you do is somewhat "taking fotographs" of the climate science webcontent. This is of course better than nothing, but it doesn't replace the "living thing". So again - I think it is important to make clear that THIS IS NO BACKUP and one shouldn't lure politics into believing that it is! By the way by looking at the CO2 curve I asked myself how one could express the whole problemacy of CO2 emissions a bit more directly. From the curve I picked three points (by handwaving read off estimation). That is in 1960 one had a CO2 microfraction of 310, 25 years later 340 and 50 years later in 2010 one had a microfraction of 410. Or in direct air fractions 0.00031, 0.00034 and 0.0004 that is 0.031, 0.034 and 0.041 per cent. If I make the ansatz that this is some curve obeying the function $$C+B e^{A*T}$$ and set 1960 to be start time (I.e. T=0) I get: $$287.5+22.5e^{0.0339 T}$$ as an approximating curve. For this you don't need big maths, just e-function and quadratic equations, i.e. undergraduate college math. According to Wikipedia Donald Trump graduated from Penn in May 1968 with a Bachelor of Science degree in economics, so this kind of math should have been part of his education i.e. eventually with a little fresh up he could in principle perform the calculations himself. So with this approximation one reaches a value of 70000 microfractions, i.e. 0.07, i.e. 7 percent air fraction at which you get already mental confusion due to CO2 poisoning within 237 years- if I haven't messed up somewhere in the calculations (please check!). Or in other words: >if CO2 emissions go on like this then even people hiding out on mountains to avoid CO2 laden cities will be poisoned within only 237 years. Again this is a calculation where you dont need big models of climate science, you just need to trust the measurements or use your own homemade measurements (which will be analogous). And yes this is only an ansatz but even if this would not be 237 years but 1000 years or worse in the other direction within the lifetime of our kids- it is clear that mankind leaves a carbon footprint which might have the sooner or later even direct health effects. Maybe one should communicate these directly to understand facts, rather then say "climate scientists calculated."
• Options
10.

Thanks, nad. It's possible HTTrack cannot do the FTP portions. I am setting up an FTP client to copy the FTP portion of CDIAC as well. Note they indicate that site was going to be migrated in September of 2017 anyway. I do not know if that migration will happen. (Who can tell?)

Again, I hope this kind of measure is completely unnecessary, but who can tell? I don't hold out much hope of communicating with the new executive staff in a convincing manner, no matter who is doing the talking or what they say. In fact, I expect some disingenuity on the listeners' part. Again, who can tell?

I also do not expect this will capture "the 'living thing'", only act as a partial safeguard against some deliberate damage and misrepresentation.

Comment Source:Thanks, nad. It's possible HTTrack cannot do the FTP portions. I am setting up an FTP client to copy the FTP portion of CDIAC as well. Note they indicate that site was going to be migrated in September of 2017 anyway. I do not know if that migration will happen. (Who can tell?) Again, I hope this kind of measure is completely unnecessary, but who can tell? I don't hold out much hope of communicating with the new executive staff in a convincing manner, no matter who is doing the talking or what they say. In fact, I expect some disingenuity on the listeners' part. Again, who can tell? I also do not expect this will capture "the 'living thing'", only act as a partial safeguard against some deliberate damage and misrepresentation.
• Options
11.

So again - I think it is important to make clear that THIS IS NO BACKUP and one shouldn't lure politics into believing that it is!

Right. I consider this obvious, but we should make it clear. I think of this as similar to running into a burning house and rescuing some precious things. It's not the same as saving the house. It's the organizational structure of the EPA, with its scientists doing their work, that's the living "house".

Comment Source:Nad wrote: > So again - I think it is important to make clear that THIS IS NO BACKUP and one shouldn't lure politics into believing that it is! Right. I consider this obvious, but we should make it clear. I think of this as similar to running into a burning house and rescuing some precious things. It's not the same as saving the house. It's the organizational structure of the EPA, with its scientists doing their work, that's the living "house".
• Options
12.
edited December 2016

I am at AGU and the speaker now talking says they generate 400 terabytes of data per day. No use saving all of it. You have to do realtime analysis and deal with it that way.

Comment Source:I am at AGU and the speaker now talking says they generate 400 terabytes of data per day. No use saving all of it. You have to do realtime analysis and deal with it that way.
• Options
13.
edited December 2016

Hi, well we are trying to save distilled products, WebHubTel. An issue very much a part of this discussion, and hopefully AGU will address, at least behind the scenes, is what happens if the Witch Hunt scenario plays out, the people who know how to operate the climate observing national assets are canned without capturing knowledge, and the control interfaces are either mothballed without care, or subjected to mischief. This can't be stopped without court orders and Congressional oversight, but, unlike military assets, there's no way these can be protected against malice. My view is what would you do if a bunch of undisciplined teenagers were let loose in control rooms. I entirely realize my imagination may be running away with me, but, again, I heard in Eric's request a plea to defend against a worst case possibility. I mean, we just don't know, but with Lamar Smith getting his way, and Tillerson as SecState and the EPA head believing there should not be one, and identifying the lists of people who contributed, I feel it is very wrong to think this "will just be okay.''

Concern is shared and summarized at: https://climatecrocks.com/2016/12/12/while-scientists-meet-trumpputin-make-lists/

Comment Source:Hi, well we are trying to save distilled products, WebHubTel. An issue very much a part of *this* discussion, and hopefully AGU will address, at least behind the scenes, is what happens if the Witch Hunt scenario plays out, the people who know how to operate the climate observing national assets are canned without capturing knowledge, and the control interfaces are either mothballed without care, or subjected to mischief. This can't be stopped without court orders and Congressional oversight, but, unlike military assets, there's no way these can be protected against malice. My view is what would you do if a bunch of undisciplined teenagers were let loose in control rooms. I entirely realize my imagination may be running away with me, but, again, I heard in Eric's request a plea to defend against a worst case possibility. I mean, we just don't know, but with Lamar Smith getting his way, and Tillerson as SecState and the EPA head believing there should not be one, and identifying the lists of people who contributed, I feel it is very wrong to think this "will just be okay.'' Concern is shared and summarized at: https://climatecrocks.com/2016/12/12/while-scientists-meet-trumpputin-make-lists/
• Options
14.

By the way, nad, your exponential curves ... 1938 ... Callendar .... https://goo.gl/JkqdkN

Comment Source:By the way, nad, your exponential curves ... 1938 ... Callendar .... https://goo.gl/JkqdkN
• Options
15.

Sound bite: Halting climate research, destroying or failing to preserve governmental climate databases, or targeting climate researchers -- any of these would amount to a declaration of war against science, and so they should be opposed by scientists across the board.

Comment Source:Sound bite: Halting climate research, destroying or failing to preserve governmental climate databases, or targeting climate researchers -- any of these would amount to a declaration of war against science, and so they should be opposed by scientists across the board.
• Options
16.
edited December 2016

Under the Freedom of Information Act, would it be illegal for the government to delete public data? If so, perhaps an injunction could be obtained now to prevent it.

Comment Source:Under the Freedom of Information Act, would it be illegal for the government to delete public data? If so, perhaps an injunction could be obtained now to prevent it.
• Options
17.

Does anyone know lawyers to whom such a question could be posed? Even if the answer is no, the question might get them thinking about how to start mounting a defense now.

Comment Source:Does anyone know lawyers to whom such a question could be posed? Even if the answer is no, the question might get them thinking about how to start mounting a defense now.
• Options
18.
edited December 2016

Jan wrote:

I heard in Eric's request a plea to defend against a worst case possibility.

Yes, exactly. This is not supposed to be a substitute for lobbying, protests, other mass actions - whatever may be required. It's just an obvious thing to do, to prepare for a worst-case scenario.

I was listening to the National Public Radio show On Point just a few minutes ago, and I was surprised to hear them discuss exactly this scenario: the mass deletion of climate databases by the Trump Administration. So it's entering the public mind, for better or worse.

Comment Source:Jan wrote: > I heard in Eric's request a plea to defend against a worst case possibility. Yes, exactly. This is not supposed to be a substitute for lobbying, protests, other mass actions - whatever may be required. It's just an obvious thing to do, to prepare for a worst-case scenario. I was listening to the National Public Radio show _On Point_ just a few minutes ago, and I was surprised to hear them discuss exactly this scenario: the mass deletion of climate databases by the Trump Administration. So it's entering the public mind, for better or worse.
• Options
19.

To this question, an attorney friend of my replied; One would think.

Comment Source:To this question, an attorney friend of my replied; One would think.
• Options
20.
edited December 2016

The reason why corruption if not destruction of databases and scientific capabilities is even countenanced is because Trump and his group don't value evidence at all. Worse, if one posits that the common connection between Putin and Tillerson and Trump is wanting to give fossil fuel energy a new lease on life, it is natural to think that it would be attractive to take measures which put off a public outcry for stringent fossil fuel emissions mitigation as far into the future as possible. Accordingly, if the science is blinded, and if the careful collections and calibrations of datasets are disrupted, if the groups and organizational knowledge within agencies and at universities is disrupted in the form of a "legal witch hunt" or lawsuits, or other kinds of intimidation, then, when large climate events begin happening, the natural caution of the sciences will say that, because of these impediments, they can no longer attribute large events to climate change, or even have a good calibration on what's happening. What happens if Operation Ice Bridge goes completely away? What happens if they don't care so much about what they see as the supposed connection of climate knowledge to improved weather forecasting? If there are cries from other countries, a Trump or a Trump lookalike can claim that, just as climate scientists pushed an "alarmist leftist campaign", other countries are espousing these positions not because they are true ("What IS Truth?"), but because they want to take advantage of U.S. business or, as Trump already claimed on the Fox News interview, to take U.S. jobs away. I have seen these arguments advanced, that the near term costs of mitigation are "just too high" in terms of economic dislocation or doing things differently. They may well seem to be. But, as Dr Katharine Hayhoe points out, arguing this is driving down a highway by looking only back through the rearview mirror. And as I illustrate, it is skeet shooting by aiming where the skeet is now, not where it will be.

Trump and company have shown blatant disregard for existing procedures and guidelines, with his personal eschewing of not only intelligence briefings, and disrespect for the intelligence community and its deep, like it or not, engineering and scientific capability, but shortcutting the standard education President-elects get about the nuclear arsenal and U.S. intelligence and armed services capabilities. Mr Trump describes these as "boring" and "not necessary." He hasn't a care about cybersecurity or cyberattacks, and probably doesn't understand them, and does not want to. Why should such a boor have ANY appreciation or care for science or academic expertise?

It will be interesting to see what the Masters of High Tech come out with, although that, too, could be just another show, like Romney, Gore, DiCaprio. And the interesting thing about a short-sighted demagogue is that he can be as much a puppet as a master.

Sorry to belabor this political stuff, but I felt the things I see and feel might be useful to put out there so people understand why I feel taking what are admittedly extreme protective measures is wise. I certainly wouldn't expect that if these observations are true to have the ability a minute past the inauguration. And there won't be any Congressional or judicial intervention right then: It'll happen too darn fast.

Comment Source:The reason why corruption if not destruction of databases and scientific capabilities is even countenanced is because Trump and his group don't value evidence at all. Worse, if one posits that the common connection between Putin and Tillerson and Trump is wanting to give fossil fuel energy a new lease on life, it is natural to think that it would be attractive to take measures which put off a public outcry for stringent fossil fuel emissions mitigation as far into the future as possible. Accordingly, if the science is blinded, and if the careful collections and calibrations of datasets are disrupted, if the groups and organizational knowledge within agencies and at universities is disrupted in the form of a "legal witch hunt" or lawsuits, or other kinds of intimidation, then, when large climate events begin happening, the natural caution of the sciences will say that, because of these impediments, they can no longer attribute large events to climate change, or even have a good calibration on what's happening. What happens if Operation Ice Bridge goes completely away? What happens if they don't care so much about what they see as the supposed connection of climate knowledge to improved weather forecasting? If there are cries from other countries, a Trump or a Trump lookalike can claim that, just as climate scientists pushed an "alarmist leftist campaign", other countries are espousing these positions not because they are true ("What IS Truth?"), but because they want to take advantage of U.S. business or, as Trump already claimed on the Fox News interview, to take U.S. jobs away. I have seen these arguments advanced, that the near term costs of mitigation are "just too high" in terms of economic dislocation or doing things differently. They may well seem to be. But, as Dr Katharine Hayhoe points out, arguing this is driving down a highway by looking only back through the rearview mirror. And as I illustrate, it is skeet shooting by aiming where the skeet is now, not where it will be. Trump and company have shown blatant disregard for existing procedures and guidelines, with his personal eschewing of not only intelligence briefings, and disrespect for the intelligence community and its deep, like it or not, engineering and scientific capability, but shortcutting the standard education President-elects get about the nuclear arsenal and U.S. intelligence and armed services capabilities. Mr Trump describes these as "boring" and "not necessary." He hasn't a care about cybersecurity or cyberattacks, and probably doesn't understand them, and does not want to. Why should such a boor have ANY appreciation or care for science or academic expertise? It will be interesting to see what the Masters of High Tech come out with, although that, too, could be just another show, like Romney, Gore, DiCaprio. And the interesting thing about a short-sighted demagogue is that he can be as much a puppet as a master. Sorry to belabor this political stuff, but I felt the things I see and feel might be useful to put out there so people understand why I feel taking what are admittedly extreme protective measures is wise. I certainly wouldn't expect that if these observations are true to have the ability a minute past the inauguration. And there won't be any Congressional or judicial intervention right then: It'll happen too darn fast.
• Options
21.
edited December 2016

...but got the following Nay from another attorney:

I applaud and share the impulse but climate data exists in so many places already that it is hard to imagine being able effectively to delete it. Furthermore, "talk" about Trump deleting it would not support an application for an injunction, and I doubt the FOIA is the the right vehicle. Unfortunately, we have to wait for at least one shoe to drop.

Comment Source:...but got the following Nay from another attorney: > I applaud and share the impulse but climate data exists in so many places already that it is hard to imagine being able effectively to delete it. Furthermore, "talk" about Trump deleting it would not support an application for an injunction, and I doubt the FOIA is the the right vehicle. Unfortunately, we have to wait for at least one shoe to drop.
• Options
22.
edited December 2016

Can you help me to analyze this response? I would like to follow up with him.

First, although sympathetic, he appears to underestimate the seriousness of the problem. Sure, climate data en mass won't be deleted, and will still exist in Europe, universities, etc. But NASA is on the leading edge of a massive data collection effort, and those databases will be under the jurisdiction of the new head.

Furthermore, these "shoes" may be dropped silently, and concurrently.

So there is the potential here, not for deleting climate data per se, but for amputating a critical branch of modern day climate science.

Comment Source:Can you help me to analyze this response? I would like to follow up with him. First, although sympathetic, he appears to underestimate the seriousness of the problem. Sure, climate data en mass won't be deleted, and will still exist in Europe, universities, etc. But NASA is on the leading edge of a massive data collection effort, and those databases will be under the jurisdiction of the new head. Furthermore, these "shoes" may be dropped silently, and concurrently. So there is the potential here, not for deleting climate data per se, but for amputating a critical branch of modern day climate science.
• Options
23.
edited December 2016

The data was collected using public funds.

I have the right to access that data under the FOIA.

Don't I have a continued right to this access?

Comment Source:The data was collected using public funds. I have the right to access that data under the FOIA. Don't I have a continued right to this access?
• Options
24.

If the FOIA is not the right vehicle, then what is?

Comment Source:If the FOIA is not the right vehicle, then what is?
• Options
25.

I think the problem is the disruption of continuity of the data. Even if an act is illegal, a department or administrator can claim ignorance, even Trump can claim ignorance and that "I've now put a stop to it." But, by that time, the damage is done. The core data, as WebHubTel points out, are too large to save. Saving that is on the order of saving the calibration data for satellites and instruments. And nad is correct, what we can save might be inconsequential for the core science. But the question is, would such an administration really be able to shut down, not only Ice Bridge, but Argo and OCO-2? As important as it is, isn't defending OCO-2 tough to do on the basis of meteorology, if one assumes there's no connection between climate and weather, as silly as that claim is?

Comment Source:I think the problem is the disruption of continuity of the data. Even if an act is illegal, a department or administrator can claim ignorance, even Trump can claim ignorance and that "I've now put a stop to it." But, by that time, the damage is done. The core data, as WebHubTel points out, are too large to save. Saving that is on the order of saving the calibration data for satellites and instruments. And nad is correct, what we can save might be inconsequential for the core science. But the question is, would such an administration really be able to shut down, not only Ice Bridge, but Argo and OCO-2? As important as it is, isn't defending OCO-2 tough to do on the basis of meteorology, if one assumes there's no connection between climate and weather, as silly as that claim is?
• Options
26.

Jan wrote in 20:

but shortcutting the standard education President-elects get about the nuclear arsenal and U.S. intelligence and armed services capabilities. Mr Trump describes these as "boring" and "not necessary."

Really? Do you have a source for that?

Comment Source:Jan wrote in 20: >but shortcutting the standard education President-elects get about the nuclear arsenal and U.S. intelligence and armed services capabilities. Mr Trump describes these as "boring" and "not necessary." Really? Do you have a source for that?
• Options
27.
edited December 2016

@nad, I do, and it's probably the Sunday New York Times, but I would have to dig to find right now, and don't have time. Need to get to work in Cambridge (MA).

Comment Source:@nad, I do, and it's probably the Sunday _New York Times_, but I would have to dig to find right now, and don't have time. Need to get to work in Cambridge (MA).
• Options
28.

@Jan I googled but couldnt find anything on that. I wondered about your comment because if I understood correctly he was asking about the use of nuclear weapons:

Donald Trump, Perhaps Unwittingly, Exposes Paradox of Nuclear Arms

Comment Source:@Jan I googled but couldnt find anything on that. I wondered about your comment because if I understood correctly he was asking about the use of nuclear weapons: Donald Trump, Perhaps Unwittingly, Exposes Paradox of Nuclear Arms
• Options
29.

The attempt to save climate data has hit the news:

I'll only quote the parts that describe these efforts, not why they're needed:

“What are the most important .gov climate assets?” Eric Holthaus, a meteorologist and self-proclaimed “climate hawk,” tweeted from his Arizona home Saturday evening. “Scientists: Do you have a US .gov climate database that you don’t want to see disappear?”

Within hours, responses flooded in from around the country. Scientists added links to dozens of government databases to a Google spreadsheet. Investors offered to help fund efforts to copy and safeguard key climate data. Lawyers offered pro bono legal help. Database experts offered help organizing mountains of data and free server space to house it. In California, Santos began building an online repository to “make sure these data sets remain freely and broadly accessible.”

[...]

That effort launched by Holthaus is one of several underway to preserve key federal scientific data.

In Philadelphia, researchers at the University of Pennsylvania, along with members of groups such as Open Data Philly and the software company Azavea, have been meeting to figure out ways to harvest and store important data sets.

At the University of Toronto this weekend, researchers are holding what they call a “guerrilla archiving” event to catalogue key federal environmental data ahead of Trump’s inauguration. The event “is focused on preserving information and data from the Environmental Protection Agency, which has programs and data at high risk of being removed from online public access or even deleted,” the organizers wrote. “This includes climate change, water, air, toxics programs.”

The event is part of a broader effort to help San Francisco-based Internet Archive with its End of Term 2016 project, an effort by university, government and nonprofit officials to find and archive valuable pages on federal websites. The project has existed through several presidential transitions.

Comment Source:The attempt to save climate data has hit the news: * Brady Dennis, [Scientists are frantically copying U.S. climate data, fearing it might vanish under Trump](https://www.washingtonpost.com/news/energy-environment/wp/2016/12/13/scientists-are-frantically-copying-u-s-climate-data-fearing-it-might-vanish-under-trump/), _Washington Post_, 13 December 2016. I'll only quote the parts that describe these efforts, not why they're needed: > “What are the most important .gov climate assets?” Eric Holthaus, a meteorologist and self-proclaimed “climate hawk,” tweeted from his Arizona home Saturday evening. “Scientists: Do you have a US .gov climate database that you don’t want to see disappear?” > Within hours, responses flooded in from around the country. Scientists added links to dozens of government databases to a [Google spreadsheet](https://docs.google.com/spreadsheets/d/12-__RqTqQxuxHNOln3H5ciVztsDMJcZ2SVs1BrfqYCc/edit#gid=0). Investors offered to help fund efforts to copy and safeguard key climate data. Lawyers offered pro bono legal help. Database experts offered help organizing mountains of data and free server space to house it. In California, Santos began building an online repository to “make sure these data sets remain freely and broadly accessible.” [...] > That effort launched by Holthaus is one of several underway to preserve key federal scientific data. > In Philadelphia, researchers at the University of Pennsylvania, along with members of groups such as Open Data Philly and the software company Azavea, [have been meeting](http://www.ppehlab.org/blogposts/2016/12/7/precarious-data-a-meeting-at-penn) to figure out ways to harvest and store important data sets. > At the University of Toronto this weekend, researchers are holding what they call a “guerrilla archiving” event to catalogue key federal environmental data ahead of Trump’s inauguration. The event “is focused on preserving information and data from the Environmental Protection Agency, which has programs and data at high risk of being removed from online public access or even deleted,” the organizers wrote. “This includes climate change, water, air, toxics programs.” > The event is part of a broader effort to help San Francisco-based [Internet Archive](https://archive.org/about/) with its [End of Term 2016](http://eotarchive.cdlib.org/2016.html) project, an effort by university, government and nonprofit officials to find and archive valuable pages on federal websites. The project has existed through several presidential transitions.
• Options
30.

An emailing from climatetruth.org:

Climate scientists have plenty of reasons to feel worried. For decades, fossil fuel interests have spread climate science disinformation and funded climate change denying politicians.

Now, science and scientific institutions face unprecedented challenges with a new administration that proposes to gut science funding, pull out of the Paris Climate Agreement, and derail the Clean Power Plan. In addition, Donald Trump has packed his cabinet with climate change deniers and fossil fuel industry magnates, including Exxon CEO Rex Tillerson for Secretary of State. But scientists won’t be silenced. This week, at the American Geophysical Union (AGU) annual conference in San Francisco, the largest gathering of Earth scientists since the election is happening — and scientists will be rallying on the streets right outside. They’ll be joined by representatives from frontline communities and allies from the climate movement. Sign the pledge to #StandUpForScience in solidarity with those rallying for a safe planet for all of us. The rally will feature some true star power: famous climate scientist Dr. Michael Mann (he created the “hockey-stick” graph of rising carbon emissions over time), Harvard professor and Merchants of Doubt author Naomi Oreskes, the executive director of Amazon Watch Leila Salazar-López, and even more climate heroes. At the rally, scientists will join indigenous groups and allies from the climate movement to call on the incoming administration to: Acknowledge that climate change is a real, human-caused, and urgent threat. Uphold the United States' commitment to the Paris Climate Agreement. Protect scientific integrity in policymaking. Protect government scientists from censorship or suppression. Reduce carbon pollution and U.S. dependence on fossil fuels. Make the U.S. a clean energy leader, and champion the just transition to a new energy era that works for all of us. If you can’t be there in person, we’ll be streaming the rally live at 3:30pm ET/12:30pm PT, on our Facebook page — so don’t miss out. When you pledge to #StandUpForScience, we will share your notes with the scientists rallying in San Francisco today. Thank you for supporting those who are doing the important work of understanding the planet we call home. Let’s work with them to protect it for generations to come. Truthfully yours, Amanda, Brandy, Emily, Brant, Daniela, and the rest of the ClimateTruth.org team P.S. After public pressure from hundreds of scientists and over 61,000 petition signers, the AGU announced that Exxon will not be a sponsor at this year's conference. This is a victory. Let’s keep the scientists’ spirits up by standing with them today.

MORE INFORMATION Facebook event: Rally to Stop the Attack on Science, December 13 at 12pm PT in San Francisco https://act.climatetruth.org/go/1459?t=9&akid=5442.298567.Y_ij0b “Trump transition team for Energy department seeks names of employees involved in climate meetings,” Washington Post, 12-09-2016 https://act.climatetruth.org/go/1474?t=11&akid=5442.298567.Y_ij0b “Trump picks Scott Pruitt, climate change denialist, to head E.P.A.,” New York Times, 12-07-2016 http://act.climatetruth.org/go/1475?t=13&akid=5442.298567.Y_ij0b “A climate denier is leading the NASA transition,” Climate Central, 12-06-2016 http://act.climatetruth.org/go/1476?t=15&akid=5442.298567.Y_ij0b “Exxon won’t sponsor AGU, a win of sorts for oil giant’s opponents,” InsideClimate News, 12-05-2016 https://act.climatetruth.org/go/1461?t=17&akid=5442.298567.Y_ij0b

Comment Source:An emailing from climatetruth.org: Climate scientists have plenty of reasons to feel worried. For decades, fossil fuel interests have spread climate science disinformation and funded climate change denying politicians. Now, science and scientific institutions face unprecedented challenges with a new administration that proposes to gut science funding, pull out of the Paris Climate Agreement, and derail the Clean Power Plan. In addition, Donald Trump has packed his cabinet with climate change deniers and fossil fuel industry magnates, including Exxon CEO Rex Tillerson for Secretary of State. But scientists won’t be silenced. This week, at the American Geophysical Union (AGU) annual conference in San Francisco, the largest gathering of Earth scientists since the election is happening — and scientists will be rallying on the streets right outside. They’ll be joined by representatives from frontline communities and allies from the climate movement. Sign the pledge to #StandUpForScience in solidarity with those rallying for a safe planet for all of us. The rally will feature some true star power: famous climate scientist Dr. Michael Mann (he created the “hockey-stick” graph of rising carbon emissions over time), Harvard professor and Merchants of Doubt author Naomi Oreskes, the executive director of Amazon Watch Leila Salazar-López, and even more climate heroes. At the rally, scientists will join indigenous groups and allies from the climate movement to call on the incoming administration to: Acknowledge that climate change is a real, human-caused, and urgent threat. Uphold the United States' commitment to the Paris Climate Agreement. Protect scientific integrity in policymaking. Protect government scientists from censorship or suppression. Reduce carbon pollution and U.S. dependence on fossil fuels. Make the U.S. a clean energy leader, and champion the just transition to a new energy era that works for all of us. If you can’t be there in person, we’ll be streaming the rally live at 3:30pm ET/12:30pm PT, on our Facebook page — so don’t miss out. When you pledge to #StandUpForScience, we will share your notes with the scientists rallying in San Francisco today. Thank you for supporting those who are doing the important work of understanding the planet we call home. Let’s work with them to protect it for generations to come. Truthfully yours, Amanda, Brandy, Emily, Brant, Daniela, and the rest of the ClimateTruth.org team P.S. After public pressure from hundreds of scientists and over 61,000 petition signers, the AGU announced that Exxon will not be a sponsor at this year's conference. This is a victory. Let’s keep the scientists’ spirits up by standing with them today. MORE INFORMATION Facebook event: Rally to Stop the Attack on Science, December 13 at 12pm PT in San Francisco https://act.climatetruth.org/go/1459?t=9&akid=5442.298567.Y_ij0b “Trump transition team for Energy department seeks names of employees involved in climate meetings,” Washington Post, 12-09-2016 https://act.climatetruth.org/go/1474?t=11&akid=5442.298567.Y_ij0b “Trump picks Scott Pruitt, climate change denialist, to head E.P.A.,” New York Times, 12-07-2016 http://act.climatetruth.org/go/1475?t=13&akid=5442.298567.Y_ij0b “A climate denier is leading the NASA transition,” Climate Central, 12-06-2016 http://act.climatetruth.org/go/1476?t=15&akid=5442.298567.Y_ij0b “Exxon won’t sponsor AGU, a win of sorts for oil giant’s opponents,” InsideClimate News, 12-05-2016 https://act.climatetruth.org/go/1461?t=17&akid=5442.298567.Y_ij0b
• Options
31.

Science emerging as a political constituency. Many had hoped that it would just be about truth, but here you have it, when truth is attempted to be thwarted.

Comment Source:Science emerging as a political constituency. Many had hoped that it would just be about truth, but here you have it, when truth is attempted to be thwarted.
• Options
32.
edited December 2016

"Truth" had always been overshadowed by politics - I just say Galilei.

Comment Source:"Truth" had always been overshadowed by politics - I just say Galilei.
• Options
33.

Between work and the underscored effort to try to protect important data assets, I could not find a Sunday Times reference to this, although there was a 2-3 page discussion of Trump's refusal to hear briefings from the IC more than once per week, and his willingness to not believe these. There was also an op-ed by former NSA head, Admiral Hayden, remarking on these developments. However, I did find:

and you might find these of limited credibility, but the day has, as Dr John has linked above, by this:

https://climatecrocks.com/2016/12/13/scientists-scramble-to-hold-off-a-new-dark-age/

Comment Source:Between work and the underscored effort to try to protect important data assets, I could not find a Sunday Times reference to this, although there was a 2-3 page discussion of Trump's refusal to hear briefings from the IC more than once per week, and his willingness to not believe these. There was also an op-ed by former NSA head, Admiral Hayden, remarking on these developments. However, I did find: http://reaction.life/trumps-bored-schoolboy-finger-nuclear-button/ https://www.google.com/amp/s/www.theatlantic.com/amp/article/509621/ and you might find these of limited credibility, but the day has, as Dr John has linked above, by this: https://climatecrocks.com/2016/12/13/scientists-scramble-to-hold-off-a-new-dark-age/
• Options
34.
edited December 2016

We’re having a lively discussion on my G+ post. Most comments have not been very practical. An exception is this by MK Taylor:

Former digital archivist here, if the groups involved with making backups of all gov data haven’t considered it, I’d strongly recommend trying to coordinate/contact the Society of American Archivists. There are groups who have been working for decades on issues related to long term archiving/preservation of digital only assets. Including scientific data. Primary SIG would be the Electronic Records Section, Metadata and Digital Objects Roundtable.

I put Nancy Beaumont, executive director of the Society of American Archivists, in touch with Bethany Wiggin, director of Penn Program in Environmental Humanities, which is leading the DataRefuge project. They are both very eager to have each other’s help, and we’re going to have a conference call on Thursday.

Comment Source:We’re having a lively discussion on [my G+ post](https://plus.google.com/u/0/117663015413546257905/posts/TJ4WCRU4DQv). Most comments have not been very practical. An exception is this by MK Taylor: > Former digital archivist here, if the groups involved with making backups of all gov data haven’t considered it, I’d strongly recommend trying to coordinate/contact the Society of American Archivists. There are groups who have been working for decades on issues related to long term archiving/preservation of digital only assets. Including scientific data. Primary SIG would be the Electronic Records Section, Metadata and Digital Objects Roundtable. I put Nancy Beaumont, executive director of the Society of American Archivists, in touch with Bethany Wiggin, director of Penn Program in Environmental Humanities, which is leading the DataRefuge project. They are both very eager to have each other’s help, and we’re going to have a conference call on Thursday.
• Options
35.
edited December 2016

It hopes to become a mirror website for US government climate data. At the help form on this website you can nominate a dataset for rescue, claim a dataset to rescue, let them know about a data rescue event, or help in some other way (which you must specify).

Since you've already been saving some data, how about contacting them?

Comment Source:Jan: I hope you know about this: * [Climate Mirror](http://climatemirror.org/) It hopes to become a mirror website for US government climate data. At the help form on this website you can nominate a dataset for rescue, claim a dataset to rescue, let them know about a data rescue event, or help in some other way (which you must specify). Since you've already been saving some data, how about contacting them?
• Options
36.

The Climate Mirror website also says:

If you want to get even more plugged into this effort, contact me and I'll add you to a Slack team for this effort.

The link leads to a form that lets you email Nick Santos, one of the people coordinating the project.

Comment Source:The Climate Mirror website also says: > If you want to get even more plugged into this effort, [contact me](http://nicksantos.com/about-and-contact/) and I'll add you to a Slack team for this effort. The link leads to a form that lets you email Nick Santos, one of the people coordinating the project.
• Options
37.

Between work and the underscored effort to try to protect important data assets, I could not find a Sunday Times reference to this, although there was a 2-3 page discussion of Trump's refusal to hear briefings from the IC more than once per week, and his willingness to not believe these.

thanks Jan for the links! The reaction article says:

This week, Greg Miller and Adam Entous at the Washington Post report that Trump has been rejecting the daily intelligence briefings offered to presidents-elect. .....So far, Trump has sat through only two, while finding time to meet with Indian business partners and TV network executives.

For Europeans it is not so easy to find out about the current status of things in the US, as you can see already at the example of the reaction article...i.e. even the English (!) get nervous about whats going on in US politics.

Apriori it seems to me that Trump is still willing to listen, but tries to get some clues from outside his presidential bubble. I see also his climate tweets a bit in this line. That is he eventually sends those out to see who is reacting how. As a business man he is probably aware of the principal-agent problem.

Comment Source:>Between work and the underscored effort to try to protect important data assets, I could not find a Sunday Times reference to this, although there was a 2-3 page discussion of Trump's refusal to hear briefings from the IC more than once per week, and his willingness to not believe these. thanks Jan for the links! The reaction article says: >This week, Greg Miller and Adam Entous at the Washington Post report that Trump has been rejecting the daily intelligence briefings offered to presidents-elect. .....So far, Trump has sat through only two, while finding time to meet with Indian business partners and TV network executives. For Europeans it is not so easy to find out about the current status of things in the US, as you can see already at the example of the reaction article...i.e. even the English (!) get nervous about whats going on in US politics. Apriori it seems to me that Trump is still willing to listen, but tries to get some clues from outside his presidential bubble. I see also his climate tweets a bit in this line. That is he eventually sends those out to see who is reacting how. As a business man he is probably aware of the principal-agent problem.
• Options
38.

For an update on the situation see:

Saving climate data (part 2), Azimuth, 15 December 2016.

Comment Source:For an update on the situation see: • [Saving climate data (part 2)](https://johncarlosbaez.wordpress.com/2016/12/16/saving-climate-data-part-2/), _Azimuth_, 15 December 2016.


text:
# Measures of association

Given a vocabulary of lexical indexing features, our central concern will be how to moderate the raw frequency counts of these features in documents (and, to a lesser extent, queries) based on {distributional} characteristics of that feature across the corpus. While we will be led to use real-valued weights to capture these relations, we begin with cruder methods, simply counting shared keywords.

The most direct way to say that a query and a document are similar is to measure the {intersection} of their respective feature sets. Let $K_q$ be the set of keyword features used in a query, and $K_d$ be the analogous set found in a document. COORDINATION LEVEL is exactly how many terms in the query overlap with terms in the document: \beq \mathname{CoordLevel}(\mathbf{q,d}) = \left| \left( K_\mathbf{q} \cap K_\mathbf{d} \right) \right| \eeq If we have many, many features and our query and documents are highly variable, then the presence of any significant overlap may be enough to identify the set of documents of interest. On the other hand, if there is a great deal of overlap between our query and many of the documents, then simply counting how big this intersection is will look like a gross measure. This fine line between one or two documents matching a query and an avalanche of thousands of matching documents occurs regularly as part of FOA.

One part of the problem is NORMALIZATION - the coordination level of intersecting features shared by both query and document seems a good measure of their similarity, but compared to what?! It's easy to show that this normalization matters. Consider the case where $\mathname{CoordLevel}(q,d) =1$, and imagine first that $K_q = K_d = 1$ also; i.e., a single word query and a one-word document. In this case the two match on the one feature they each have. Intuitively, this is a perfect match; you couldn't do any better. But now imagine that the query includes 10 keywords, the document contains 1000, but still $\mathname{CoordLevel}(q,d) =1$. The same intuition suggests that this should be judged a much poorer match, but our measure does not reveal this. Unless we normalize by something that reflects how many other features we might have matched, we can't have a useful measure of association.

One natural normalizer is to take the {average} of the number of terms in the query and in the document and compare the size of the intersection to it. This gives us the Dice coefficient: \beq \mathname{Sim}_{Dice}(\mathbf{q,d}) = 2 \frac{\left| \left( K_\mathbf{q} \cap K_\mathbf{d} \right) \right|} {\mid K_\mathbf{q} \mid + \mid K_\mathbf{d} \mid} \eeq The average number of features may or may not be appropriate, again depending on what a typical query and typical document is. Often a document has many, many keywords associated with it while queries may have only one or two. This average is not a very good characterization of either.

Another perspective on similarity says that missing features are as significant as shared ones. The SIMPLE MATCHING COEFFICIENT gives equal weight to those features {\em included} in both query and document and those {\em excluded} from both, and normalizes by the total number of keyword features $NKw$: \beq \mathname{Sim}_{simple}(\mathbf{q,d}) = \frac{\left| \left( K_\mathbf{q} \cap K_\mathbf{d} \right) \right| + \left| \left( (NKw - K_\mathbf{q}) \cap (NKw - K_\mathbf{d}) \right) \right|} {\mid NKw \mid} \eeq

FOA © R. K. Belew - 00-09-21

format: plain


text:
# Am I in the correct direction for solving this problem?

Virtual call centers are staffed by individuals working out of their homes. An airline is considering employing home agents, but only if their level of customer satisfaction is greater than $80\%$. A test was conducted with home servicing agents. In a sample of size 300 customers, 252 reported that they were satisfied with the service

A. The airline set up the test: \begin{align*} H_0 &: p \leq 0.8 \\ H_1 &: p > 0.8 \end{align*} at level $\alpha = 0.05$ ; p is the true proportion among all customers who will be satisfied by service from home agent

Set up and sun this test, using significance level $$\alpha =.05$$. What conclusion do you reach?

\begin{align*} n = 300&, N =252 \\ H_0 &: p \leq 0.8 \\ H_1 &: p > 0.8 \end{align*}

Test statistic: One Group Proportion

$$Z = \frac{.84 - .8}{\sqrt{.8(1-.84)/300}}$$

From here I would compare it with the Z value at .05?


text:
GMAT Changed on April 16th - Read about the latest changes here

 It is currently 27 May 2018, 22:01

### GMAT Club Daily Prep

#### Thank you for using the timer - this advanced tool can estimate your performance and suggest more practice questions. We have subscribed you to Daily Prep Questions via email.

Customized
for You

we will pick new questions that match your level based on your Timer History

Track

every week, we’ll send you an estimated GMAT score based on your performance

Practice
Pays

we will pick new questions that match your level based on your Timer History

# Events & Promotions

###### Events & Promotions in June
Open Detailed Calendar

# A sporting goods store has a section that displays baseball caps, and

Author Message
TAGS:

### Hide Tags

GMAT Forum Moderator
Joined: 28 May 2014
Posts: 523
GMAT 1: 730 Q49 V41
A sporting goods store has a section that displays baseball caps, and [#permalink]

### Show Tags

17 Nov 2017, 09:21
2
This post was
BOOKMARKED
00:00

Difficulty:

15% (low)

Question Stats:

81% (01:07) correct 19% (02:30) wrong based on 74 sessions

### HideShow timer Statistics

A sporting goods store has a section that displays baseball caps, and each shelf in the section displays 10 caps. After 16 caps were sold, each shelf was left with 7 caps except for the final shelf, which held 9 caps. How many shelves are in the section displaying baseball caps?

(a) 3
(b) 5
(c) 6
(d) 7
(e) 11

_________________
Director
Joined: 14 Nov 2014
Posts: 667
Location: India
Schools: Rotman '20 (S)
GMAT 1: 700 Q50 V34
GPA: 3.76
Re: A sporting goods store has a section that displays baseball caps, and [#permalink]

### Show Tags

17 Nov 2017, 09:59
10n - (7(n-1) +9) = 16
N= 6
SC Moderator
Joined: 22 May 2016
Posts: 1681
A sporting goods store has a section that displays baseball caps, and [#permalink]

### Show Tags

17 Nov 2017, 10:11
saswata4s wrote:
A sporting goods store has a section that displays baseball caps, and each shelf in the section displays 10 caps. After 16 caps were sold, each shelf was left with 7 caps except for the final shelf, which held 9 caps. How many shelves are in the section displaying baseball caps?

(a) 3
(b) 5
(c) 6
(d) 7
(e) 11

Before caps were sold there were 10 caps per shelf. (6*10) = 60 caps
16 are sold. (60-16) = 44 remain

After the caps are sold, there are 7 caps on all but the last shelf, which has 9.

So remaining caps are:
1) 7 caps on 5 shelves; and
2) 9 caps on 1 shelf

1) 5 * 7 = 35
2) 35 + (9)(1) = 44

_________________

In the depths of winter, I finally learned
that within me there lay an invincible summer.

Math Expert
Joined: 02 Aug 2009
Posts: 5784
Re: A sporting goods store has a section that displays baseball caps, and [#permalink]

### Show Tags

17 Nov 2017, 10:14
1
KUDOS
Expert's post
saswata4s wrote:
A sporting goods store has a section that displays baseball caps, and each shelf in the section displays 10 caps. After 16 caps were sold, each shelf was left with 7 caps except for the final shelf, which held 9 caps. How many shelves are in the section displaying baseball caps?

(a) 3
(b) 5
(c) 6
(d) 7
(e) 11

algebra...
Let the number be x, so total = 10x
after taking out 16, x-1 have 7 each and 1 has 9..
so $$10x-16=7(x-1)+9........10x-16=7x-7+9.....3x=16+9-7=18...x=6$$

another way..
if you remove 16, it leads to 3 less in x-1 and 1 less in last one..
so $$16= 3(x-1)+1.....16=3x-3+1.....3x=18...x=6$$

C
_________________

Absolute modulus :http://gmatclub.com/forum/absolute-modulus-a-better-understanding-210849.html#p1622372
Combination of similar and dissimilar things : http://gmatclub.com/forum/topic215915.html

GMAT online Tutor

Target Test Prep Representative
Affiliations: Target Test Prep
Joined: 04 Mar 2011
Posts: 2442
Re: A sporting goods store has a section that displays baseball caps, and [#permalink]

### Show Tags

27 Nov 2017, 12:14
saswata4s wrote:
A sporting goods store has a section that displays baseball caps, and each shelf in the section displays 10 caps. After 16 caps were sold, each shelf was left with 7 caps except for the final shelf, which held 9 caps. How many shelves are in the section displaying baseball caps?

(a) 3
(b) 5
(c) 6
(d) 7
(e) 11

Since 16 caps were sold, we can say that 3 were removed from 5 shelves and 1 was removed from the 6th shelf. So there are 6 total shelves.

Alternate Solution:

We can let n = the number of shelves. Thus the original number of caps is 10n. After 16 caps were sold, n - 1 shelves have 7 caps each and the last shelf has 9 caps; so we can create the following equation:

10n - 16 = 7(n - 1) + 9

10n - 16 = 7n - 7 + 9

3n = 18

n = 6

_________________

Jeffery Miller

GMAT Quant Self-Study Course
500+ lessons 3000+ practice problems 800+ HD solutions

Re: A sporting goods store has a section that displays baseball caps, and   [#permalink] 27 Nov 2017, 12:14
Display posts from previous: Sort by


text:
# If the feasible region for a LPP is __________,then the optimal value of the objective function $Z=ax+by$ may or may not exist.

Toolbox:
• If $R$ is unbounded ,where $R$ is the feasible region ,then a maximum or a minimum value of the objective function may not exist.
Step 1:
If the feasible region for a LPP is unbounded, then the objective function $Z=ax+b$ may or may not exist.
Step 2:
Unbounded means that the feasible region does extend indefinitely in any direction.


text:
# Example of a compact set whose set of limit points is countably infinite.

I need to find an example of a compact set whose set of limit points is countably infinite.

-

HINT: Start with the set $A=\{0\}\cup\left\{\frac1n:n\in\Bbb Z^+\right\}$; that’s a compact set with one limit point. Now add to $A$ a sequence converging to each of the points $\frac1n$. You can do this in $\Bbb R$, but it’s a little easier to describe and visualize if you first embed $A$ in $\Bbb R^2$ and then let the new sequence converge vertically to the points $\langle\frac1n,0\rangle$.

-
Excellent, thanks a lot. –  Jorge Dec 7 '12 at 19:51
@Jorge: You’re welcome. –  Brian M. Scott Dec 7 '12 at 19:55

Hint: In $\mathbb{R}$ we want the set to be closed and bounded. Make a set whose limit points are, say, $0$ and $1,1/2,1/3,1/4,1/5,\dots$.

To make a set with the right limit points, you might start with making a set whose only limit points are $1$, $2$, $3$, and so on. (This is of course not compact.) Then produce the desired set by using the reciprocal, not forgetting the reciprocal of "$\infty$."

-
Excellent, thank you. –  Jorge Dec 7 '12 at 19:50

$$A = \{ (1/m, 1/n),(1/m,0),(0, 1/n) : m , n \in \mathbb{N} \}$$ The set of limit point of $A$ is $\{(1/m,0) : m \in \mathbb{N} \} \cup \{ (0, 1/n) : n \in \mathbb{N} \}$ is contable infinite and $A$ is compact because it is bounded and closed.

-


text:
# Mister And Mississippi - Meet Me At The Lighthouse

November 12, 2015

## Lyrics

Meet me at the lighthouse
Bye the seaside, when the night falls
I'll wait there, staring out to sea
Let's sail away now, disappear with me

It's you, and the things that you do
I can be myself around you
For the first time I knew
That it's always been you

We set sail, as the waves break,
on the seaside, near the lighthouse
We look back, see the lights fade
And in that moment, all goes quiet

It's you, and the things that you do
I can be myself around you
For the first time I knew
That it's always been you

All I am, it's all for you
All I am, it's all for you
All I am, it's all for you
All I am, all I am, for you

All I am, it's all for you
All I am, it's all for you
All I am, it's all for you
All I am, all I am, all I am, for you


text:
# Why does the logistic model for binary logistic regression return the probability that the outcome was in class/category 1?

In the context of simple binary logistic regression (https://en.wikipedia.org/wiki/Logistic_regression) we have $$p(x)=\frac{1}{ 1 + e^{\beta x}}$$, where $$p(x)$$ is interpreted as the probability that the observation $$x$$ has $$y=1$$, i.e. $$x$$ belongs to category/class 1.

My question is: Why does the fitted model return the (conditional) probability for category/class 1; why not category/class 0 instead? Or differently asked: How can one determine whether the fitted model returns the (conditional) probability of category/class 1 as opposed to category/class 0?

I would like to see answers that show where in the defintion of the logistic model the assumption that $$p(x)= p(y=1 \mid x)$$ is made explicit.

Binomial distributions are defined to give the probability of class $$1$$ instead of class $$0$$.
Yes, the theory could be tweaked to do the reverse, but there is no benefit to doing so (since all of the information about the probability of being in one class is contained in the probability of being in the other class), so we stick with the convention of a binomial distribution pertaining to the probability of class $$1$$ instead of class $$0$$.
• Okay, then it is just convention to work with the probability of class 1. I know that if the number of trials is 1 ($n = 1$) the Binomial distribution becomes a Bernoulli distribution. However, I do not understand why you refer to the binomial distribution? Or differently asked: What are use cases for logistic regressions where $n \neq 1$, i.e., $n > 1$? Sep 26 at 16:23
• @CausalQuestions Sure, call it Bernoulli. // Your $n> 1$ question warrants it’s own post.


text:
• S Biswas

Articles written in Pramana – Journal of Physics

• Dual pole eikonal ands-channel singularities

Eikonalization of dual pole amplitudes, such as the Veneziano amplitude, is shown to lead to singularities in the impact parameter plane, which, in the eikonal approximation, may be interpreted as branch points in the direct channel angular momentum plane. This result is discussed in the light of dual absorption models.

• 3He-rich solar flares

A new subgroup of3He rich solar flares is found on reanalysing the global data.3He/H ratio as a function of maximum proton flux at an energy of about 10 MeV shows a break-up of the data into two groups. The first group follows the anticorrelation of3He/H ratio with the proton flux, as expected in the plasma process acceleration models. But the second group has a constant3He/H ratio as a function of maximum proton flux. This is not in conformity with the plasma process models. But this is expected in models where the nuclear spallation reactions are responsible for the production of3He. It is also found that the same break-up into two distinct groups follows if one plots the location of the flares in the solar disc. The first group is more or less confined to the west limb of the Sun, whereas the second group is more widely spread out across the solar disk.

• Ionization states of cosmic rays:Anuradha (IONS) experiment in Spacelab-3

The measurements of the ionization states, composition, energy spectra and spatial distribution of heavy ions of helium to iron of energies 10–100 MeV/amu in the anomalous cosmic rays are of major importance in understanding their origin which is unknown at present.Anuradha (IONS) cosmic ray experiment in Spacelab-3 was designed to determine the above properties in near earth space and this had a highly successful flight and operations aboard the shuttle Challenger at an orbital altitude of 352 km during 29 April to 6 May 1985. The instrument employs solid state nuclear track detectors (CR-39) of high sensitivity and large collecting area of about 800 cm2 and determines the arrival time information of particles with active elements. Experimental methods, flight operations and preliminary results are briefly described. Initial results indicate that relatively high fluxes of low energy cosmic ray α-particles, oxygen group and heavier ions were obtained. The flight period corresponded to that of quiet Sun and the level of solar activity was close to solar minimum. It is estimated that about 10,000 events of low energy cosmic ray alpha particles with time annotation are recorded in the detector together with similar number of events of oxygen and heavier ions of low energy cosmic rays.

• Confinement of tensor gluons — a classical approach

We look at the confinement of tensor gluons (fμν(c) field) in a strong gravity background and find that the strong gravity provides a trap for the confinement of colour waves of selected frequencies. We assume that the tensorfμν(c) field (mediating quanta: tensor 2+f-meson) satisfies Einstein-like equations with a cosmological constant. The colour field satisfy equations resembling Maxwell form of the linear theory of gravitation and see the effect offμν(c) field as playing the role of a medium having space dependent dielectric permeabilities. The solution of colour field equations resemble harmonic oscillator type wave functions with equispaced energy levels (no continuum) leading to confinement.

• Confinement model for quarks

A confinement model of hadron with its constituent quarks bound in a strong gravitational field is presented. The gravitational field plays the role of a medium having, as if, space dependent permeabilities from a fixed centre. The massless Dirac equation modified by the gravitational field is solved. The solution for the wavefunction of the quarks obtained shows the characteristic features of confinement, i.e., (i) wavefunction with higher energy states lying closer to the centre, (ii) equispaced energy levels without continuum, (iii) the quark orbits lying within a distance ∼ 10−14 cm, the characteristic radius of a typical hadron.

• Model of confinement

Confinement model for quarks and gluons is formulated. An attempt has been made to derive the dielectric function starting from a Lagrangian that also determines the quark and gluon field equations. Deconfinement mechanism is also discussed.

• Aharonov-Bohm effect from five dimensional space-time

We present an alternative non-quantum mechanical description of Aharonov-Bohm effect.

• Dirac equation in time dependent electric field and Robertson-Walker space-time

We show that the motion of a Dirac electron in a time dependent electromagnetic field can be considered as a motion in a dielectric medium with time dependent dielectric function. We find that this electromagnetic case is analogous to the description in Robertson-Walker (RW) space-time. We solve the Dirac equation is such a simulated space-time.

• Geometric interpretation of hadronization model

A hadronization model termed as geometric dielectric confinement model is described. The model describes the charmed meson decays quite successfully. In the model we assume that the non-abelian gauge field describing the colour force simulates the effect of a medium having space-dependent dielectric constant. The quarks produced in weak decays move in the dielectric medium such that they are free in limited region of space (r⋍0) and cannot appear as asymptotic states resulting in hadronization. It is found that the dielectric medium resembles anti-desitter microuniverse and the quarks behave essentially as free particles damped by gaussian distribution. The model reproduces from a single Lagrangian the quark motion as well as the form of dielectric function.

• Regge relation and geometrization of fundamental constant

We show that the Regge relation between angular momentum and mass is due to curved space time description of basic interactions. We try to understand the geometrization ofh ande in the light of the relation.

• WKB approximation in complex time

The WKB approximation to the one particle Schrödinger equation in time is used to obtain the wavefunction at a given point as a sum of semiclassical terms, each corresponding to a different classical trajectory (real or complex) but ending up at the same point. A method to find out reflection coefficient for processes involving one and two turning points is developed and it is shown that the semiclassical complex analysis reproduces exactly the reflection coefficient that is obtained through the exact solution of the problem. The connection between pair production and reflection amplitude is also shown. The pair production amplitude in a time dependent gravitational background is calculated and it is shown that the vacuum considered in complex trajectory WKB analysis refers to adiabatic vacuum.

• WKB approximation in complex time II

The non-perturbative method, developed recently, of WKB approximation in complex time is applied to some known curved space time. Three cases namely (1) static in and out region, (2) non-static in and out region, (3) static in and non static out region are considered here. We find non-trivial particle production corresponding to the quantum vacuum definition of Castagnino and Mazzitelli.

• Particle production in curved spacetime

Particle production in curved spacetime has been discussed through the method of complex time WKB approximation. We consider Dirac equation in non-flat spacetime to understand particle production as particle-antiparticle rotation. The method is also generalized to understand particle production through parametric resonance. To understand the method of CWKB we consider particle production in Kasner spacetime as an example.

• Dirac current in expanding spacetime

We study the behaviour of Dirac current in expanding spacetime with Schrödinger and de Sitter form for the evolution of the scale-factor. The study is made to understand the particle-antiparticle rotation and the evolution of quantum vacuum leading to particle production in such spacetime.

• Starobinsky model in Schrödinger description

In the Starobinsky inflationary model inflation is driven by quantum corrections to the vacuum Einstein equation. We reduce the Wheeler-DeWitt equation corresponding to the Starobinsky model to a Schrödinger form containing time. The Schrödinger equation is solved with a Gaussian ansatz. Using the prescription for the normalization constant of the wave function given in our previous work, we show that the Gaussian ansatz demands Hawking type initial conditions for the wave function of the universe. The wormholes induce randomness in initial states suggesting a basis for time-contained description of the Wheeler-DeWitt equation.

• Stability study and time resolution measurement of straw tube detectors

Straw tube detectors are single wire proportional counters that are widely used as tracking devices. We have carried out R&D activity with a straw tube detector prototype. The aim of this work is to study the stability of the performance in terms of gain and energy resolution of these types of detectors under high rate of radiation. The gain and energy resolution of the detector are studied along with its variation with ambient temperature and pressure. X-rays from a radioactive source are used to irradiate the detector and to monitor the energy spectra simultaneously for calculating the gain. The method followed here is unique as the ageing measurements have been performed without using an accelerated particle beam or any radiation generator. The performance of a straw tube detector has been studied in a laboratory for more than 800 h at a stretch using a single sealed radioactive X-ray source with high intensity. Variation of gain and energy resolution of the straw tube detector under X-ray irradiation in Ar/CO$_2$ gas mixture (volume ratio 80/20) is discussed in this article. The gain of an aged straw depends on gas flow rate. We have estimated the time required for the gain of a straw tube detector under ageing tests to recover on increasing the gas flow rate. We have also estimated the time resolution of the straw tube detectors by using cosmic rays as the trigger for the Ar/CO$_2$ gas mixture in 70/30 volume ratio (different gas mixture was used for the measurement of the time resolution). This type of ageing and time resolution measurements in Ar/CO$_2$ gas mixture has not been reported earlier. The details of the measurement process and the experimental results are presented in this article.

• Cosmic ray flux and lockdown due to COVID-19 in Kolkata – Any correlation?

Cosmic ray muon flux is measured by the coincidence technique using plastic scintillation detectors in the High Energy Physics Detector Laboratory at Bose Institute, Kolkata. Due to the COVID-19 outbreak and nationwide complete lockdown, the laboratory was closed from the end of March 2020 till the end of May 2020. After lockdown, although the city is not in its normal state, we still were able to take data on some days. The lockdown imposed a strict restriction on the transport service other than the emergency ones and also most of the industries were shut down in and around the city. This lockdown has significant effect on the atmospheric conditions in terms of change in the concentration of air pollutants. We have measured the cosmic ray flux before and after the lockdown to observe the apparent change if any, due to change in the atmospheric conditions. In this article, we report the measured cosmic ray flux at Kolkata (22.58°N 88.42°E and 11 m above the Sea Level) along with the major air pollutants present in the atmosphere before and after the lockdown.

• # Pramana – Journal of Physics

Volume 96, 2022
All articles
Continuous Article Publishing mode

• # Editorial Note on Continuous Article Publication

Posted on July 25, 2019


text:
# Dream Team – combining Tableau and R

November 3, 2013
By

(This article was first published on Data * Science + R, and kindly contributed to R-bloggers)

Last quarter was a bit too busy to write some new blog post because of a new job. And changing the job often come along with changing the tools you work with. That was my way to Tableau. Tableau is one of the new stars in the BI/Analytics world and definitely worth a look. The people at Tableau describe their tool as an instrument that combines interactive visual analytics, sophisticated visualization methods, an easy to use graphical user-interface and a lot of connectors for dozens of different data sources ranging from standard Excel to more “Big Data” like things such as different Hadoop distributions and NoSQL databases. What is really impressing is that you can create interactive visualizations in just no time.

I’m currently using Tableau to carry out explorative analysis and reporting up to management level (creating high polish PDFs). I’m quite satisfied with Tableaus capabilities for this even that Tableaus focuses more on Self-BI. What I missed are sophisticated analytical methods like different algorithms for classification, regression, clustering et cetera. So I was really happy that Tableau announced the integration of R in their upcoming release and quickly enrolled in their beta tester program a few weeks ago. Time now to share some insights from my first experiments with the beta version of the upcoming Tableau release 8.1.

First step is to get the connection between R and Tableau based on Rserve up and running. Following the approach given here it should be really easy even for Windows. It is good to know that you can launch Rserve with “R CMD Rserve_d” instead of “R CMD Rserve” to grab some information from the communication between Tableau and R for debugging (here you can find another short comment about some more advanced debugging techniques). Alternatively you can launch Rserve from within R with:

library(Rserve)
Rserve()

If everything works fine you can test you connection from inside Tableau by selecting “Help -> Manage R Connections …”. Be sure to replace “localhost” with “127.0.0.1” because of a bug in the current beta version (tested with 8.1 Beta 2 - may be fixed in newer versions) which will slow down performance otherwise.

The integration uses so called table calculations to retrieve results from calculations within R. A table calculation could be used to state a calculation like “Partition the sales data by region and calculate the overall revenue per product category”. This means that the arguments from Tableau to R have to be specified as aggregates. The result of the R calculation then should be one value or the same number of values as in the partition (vector). This sounds a little bit difficult and to be honest in the beginning it was difficult for me to understand the logic behind. But let’s try some examples to demonstrate what that means.

The first one is derived from the official Tableau blog and demonstrates the application of k-Means using the iris dataset.

Our aim is visualize the resulting clustering which should give us some understanding about how it matches the true classification. We use a calculated field to pass the data to R and get the resulting clustering ids from the k-Means function. The code for the table calculation function inside the calculated field is very simple:

SCRIPT_INT('result <- kmeans(x = data.frame(.arg1,.arg2,.arg3,.arg4), 3)
result$cluster', SUM([Sepal#Length]),SUM([Sepal#Width]),SUM([Petal#Length]),SUM([Petal#Width]) )  SCRIPT_INT( R code, Parameters ) is the Tableau table calculation for running R code that returns an integer value or vector (Tableau also offers functions that return boolean, string and real values). It is important to know, that this function only returns one value/vector and such, if you need more parts of the results from your R computation, you have to create additional calculated fields. In the code above we use data.frame on the given parameters (going from .arg1 to .argN where N is the number of parameters) first to create a data object for the k-Means function. The resulting clustering ids are then passed back to Tableau. As you see all four parameters must be aggregated first (SUM(...)) before we can use them in our R calculation. We can then use the calculated field to show the cluster assignment by assigning every cluster id a different shape. Because we want to see the results on a detailed level we need to deactivate the default aggregation in Tableau by clicking on “Analysis -> Aggregate Measures”. The resulting plot reveals that there is a huge mismatch between the clusters and classes - not the best clustering solution. Now it would be nice to strengthen the interactive features of Tableau and therefore parameterize the R calculations in a way that we can modify the calculation from within Tableau. This leads to my second example visualizing the results from a PCA - showing a scatterplot of the objects in the space of the two choosen components. Therefore we use the USArrets dataset (Violent Crime Rates by US State) and the standard R function princomp for doing principal component analysis: SCRIPT_REAL('ncomp <- max(.arg5) df <- data.frame(.arg1,.arg2,.arg3,.arg4) pc <- princomp(df, cor = TRUE) as.numeric(pc$scores[,ncomp])',
SUM([Assault]),SUM([Murder]),SUM([Rape]),SUM([UrbanPop]),[No. Component - x axis]
)

We can extract the scores for every component from the result of princomp. Because we need the scores for x and y axis we create two calculated fields - one for each axis (above you can see the code for the calculated field that shows the x axis). Together with the data we can pass a parameter from Tableau to R that determines which component scores we want (one parameter for every calculated field).

Because R retrieves this parameter the same size as the other inputs we need to aggregate the parameter to yield a scalar value for selecting the right scores. Now it’s beautiful to see how we can integrate these R calculations into Tableau’s typical interactive workflow: Drag one of the attributes to column and one to ‘Shape’ - now you can see how theses features load on the selected components. If you activate the parameter control for each of the parameters you can change the scores mapped on x and y axis on the fly. Really a nice tool to explore the results of a principal component analysis.

My last example demonstrates a use case that is common for me but that in my opinion also shows the limitations of the current R integration. Imagine that your personal data scientist sends you a fine tuned data mining model that you want to use for further prediction. You want to apply these model on your current dataset in Tableau and use the predictions as additional feature for your visualizations to answer further questions. How could the new R integration in Tableau helps us with that? Let’s start and first create the model outside of Tableau because this is likely the way the data scientist will do it (using tools from his own development environment). For this we create a simple classification model with rpart (decision trees) for predicting the outcome of the species feature from our iris dataset and save the model in an RData file ready for sharing (we could also use a more standardized format like PMML available in the corresponding package pmml).

require(rpart)
data(iris)
rpartModel <- rpart(Species ~ ., data = iris)
save(rpartModel, file = "C:/tmp/RpartModel.RData")

Now we could try to utilize the model from within Tableau. For this we create the following calculated field that loads the model and predicts the outcome of the given data. The code looks like this

SCRIPT_STR('
require(rpart)
data(iris)
newiris <- data.frame(Sepal.Length = .arg1,
Sepal.Width = .arg2,
Petal.Length = .arg3,
Petal.Width = .arg4)
as.character(predict(rpartModel, newiris, type ="class"))',
ATTR([Sepal#Length]),ATTR([Sepal#Width]),ATTR([Petal#Length]),ATTR([Petal#Width])
)

It is not surprising that the visualization of true class (color) versus predictions (shape) reveals good results. As before you need to deselect “Aggregate Measures” for the plot.

So far so good. But what if we want to use the predictions together with the class labels to create a simple confusion matrix? Let’s try and plot both features against each other and color the results using an additional calculated field called Misclassification (IF ATTR([Species]) = [R Prediction] THEN 'No' ELSE 'Yes' END). What I achieved after some experiments are either disaggregated results (showing a “1” for each entry colored according to “Misclassification”“ - which is not very helpful) or a single prediction for each partition of the aggregated data that gets multiplied by the number of data points inside each partition. Here are two pictures showing both possible results:

Sadly, until now I didn’t get the things running in way it should be (maybe some of you have an helpful idea idea - if so then don’t hesitate to leave a comment) which make me think if some of my desired use cases are not possible with the current R integration in Tableau.

Finally I want to summarize what I observed so far: I’m really excited whenever a commercial software company announces that it will integrate R into their software stack. What Tableau did with R is really useful as the shown uses cases hopefully demonstrated. But the integration is also not that simple than other things in Tableau and I still need more time to get a better picture of things that work and things that won’t work. Therefore it is perfect, that Tableau has a very active community and I’m sure that given some weeks we will see dozens of new possibilities on how this integration of R can produce fruitful results.

Please also find my workbook with all the examples from this blog post attached: Test_R_Integration_Tableau_8_1.twbx


text:
# scipy.sparse.linalg.expm_multiply¶

scipy.sparse.linalg.expm_multiply(A, B, start=None, stop=None, num=None, endpoint=None)[source]

Compute the action of the matrix exponential of A on B.

Parameters: A : transposable linear operator The operator whose exponential is of interest. B : ndarray The matrix or vector to be multiplied by the matrix exponential of A. start : scalar, optional The starting time point of the sequence. stop : scalar, optional The end time point of the sequence, unless endpoint is set to False. In that case, the sequence consists of all but the last of num + 1 evenly spaced time points, so that stop is excluded. Note that the step size changes when endpoint is False. num : int, optional Number of time points to use. endpoint : bool, optional If True, stop is the last time point. Otherwise, it is not included. expm_A_B : ndarray The result of the action $$e^{t_k A} B$$.

Notes

The optional arguments defining the sequence of evenly spaced time points are compatible with the arguments of numpy.linspace.

The output ndarray shape is somewhat complicated so I explain it here. The ndim of the output could be either 1, 2, or 3. It would be 1 if you are computing the expm action on a single vector at a single time point. It would be 2 if you are computing the expm action on a vector at multiple time points, or if you are computing the expm action on a matrix at a single time point. It would be 3 if you want the action on a matrix with multiple columns at multiple time points. If multiple time points are requested, expm_A_B[0] will always be the action of the expm at the first time point, regardless of whether the action is on a vector or a matrix.

References

 [R233] Awad H. Al-Mohy and Nicholas J. Higham (2011) “Computing the Action of the Matrix Exponential, with an Application to Exponential Integrators.” SIAM Journal on Scientific Computing, 33 (2). pp. 488-511. ISSN 1064-8275 http://eprints.ma.man.ac.uk/1591/
 [R234] Nicholas J. Higham and Awad H. Al-Mohy (2010) “Computing Matrix Functions.” Acta Numerica, 19. 159-208. ISSN 0962-4929 http://eprints.ma.man.ac.uk/1451/

#### Previous topic

scipy.sparse.linalg.expm

#### Next topic

scipy.sparse.linalg.onenormest


text: Figure 1: (Left) Perturbative quantum chromodynamics (Feynman diagram) view of the energy loss amplitude for a bottom quark jet moving toward the left in a quark-gluon plasma that radiates a few gluons. This takes place in our usual flat four-dimensional spacetime. (Right) The dual holographic anti-de Sitter view of a rightward moving supersonic bottom quark jet modeled as string drag in a curved ten-dimensional $AdS5×S5$ spacetime equipped with a flat three-dimensional black hole. The energy loss can be observed as energy stress shock waves and sound waves in the flat four-dimensional boundary on top. The black hole horizon, shown displaced in the depths of the fifth dimension, is the geometric analog of the hot uniform plasma. The bottom quark is represented as a string with one end attached to a higher dimensional “$D7$” brane between the horizon and the boundary (not illustrated).