text:
# One More Note on Quantum Teleportation

Yesterday, I posted a detailed explanation of the technique of quantum teleportation and hopefully demonstrated that the process is not as complicated as one might expect! And also that it is not going to lead to Star Trek transporters anytime soon. No, unfortunately, we will have to settle for a mere quantum Internet instead.

But even that technology has its hurdles to overcome, one being the problem of transporting an entangled qubit to the destination you would like to send your message. As those who have been following along all know, entanglement is a quantum state. Quantum states, it turns out, can be quite fickle; they are very easy to disrupt! In previous posts, I have loosely (and carelessly) thrown around the concept of collapsing a wave function. (I am fairly confident that I linked to the Wikipedia article on wave function collapse somewhere, but alas I cannot find it!) What that expression means is that if you have a quantum object whose wave function is a superposition (read: combination) of different states, whenever that object is "observed," its wave function collapses down to one of the possible states. From that point on, any other time the object is measured, the probability of finding it in that one state is $$100\%$$—in other words, it's the only state possible until the object's wave function changes again.

As an example, think back to our quantum playing cards, which are described by the wave function$$\lvert \psi \rangle = \frac{1}{\sqrt{2}} \left( \lvert \spadesuit \rangle + \lvert \diamondsuit \rangle \right) \label{040817:qpc}$$as long as they are face down. At the instant the card is turned face up, its wave function collapses to either\begin{equation*} \lvert \psi \rangle = \lvert \spadesuit \rangle, \label{270717:qpcs} \end{equation*}or\begin{equation*} \lvert \psi \rangle = \lvert \diamondsuit \rangle. \label{270717:qpcd} \end{equation*}So long as the card remains face-up, its wave function continues to be one of those two options, and the card's value does not change. Only when the card is turned face down is its wave function reset to the original state. (Note: there is nothing in quantum mechanics which says the card's wave function must reset when it is turned face down; that's simply the way I programmed it for the sake of demonstration.)

Now, this fact seems to imply that there is something special about "observers" in quantum mechanics, and much ink is spilled by nonexperts who take that expression and run with it off into the realm of nonsense. However, in physics, an "observer" is anything that interacts with that quantum object, including other objects. Say our object is a sub-atomic particle, and it collides with another particle. That interaction can change our particle's wave function. If our particle is in an entangled state and it collides with another particle, the collapse of our particle's wave function destroys its entanglement. If our particle is no longer entangled, then we cannot use it to teleport a qubit! Needless to say, we do not want this to happen, and the best way to prevent it is by isolating our particle from all others.

However, we then run into the problem that everything in the Universe is made out of particles! The wires or optical fibers that we might use to relocate our entangled qubits? Those are made out of particles, and when our qubits collide with these particles, their wave functions collapse and we lose entanglement. What if we shoot our entangled particles through the air? Well, that too is made out of particles, and colliding with any of them destroys our entanglement! In fact, to minimize the risk of collisions, we must send our entangled particles through a vacuum, and no turbomolecular pump in the world can pull a stronger vacuum than that which exists naturally in outer space! Therefore, space is a natural setting in which to place the infrastructure of a quantum Internet, but even that will do us no good if we cannot access it from the surface of Earth.

Therein lies the true significance of the Chinese team's experiment. They are not testing whether or not quantum teleportation is possible, as it has been done many times before. They are setting a new distance record, but so far that has only been limited by how far one can separate one's particles before losing entanglement—in a pure vacuum, this distance is infinite. Instead, the significant result here is the uplink between an entangled particle on the ground and one in orbit, which allows a qubit to be sent to an environment in which it can easily be transferred around the world. One of the more technically challenging aspects of this procedure is that it requires incredibly precise aim. Teleporting qubits between entangled particles in different locations? That's easy! Getting one of those particles to its desired location on board a satellite traveling thousands of miles per hour hundreds of miles overhead? Now, that is hard! Here is how they pulled it off with lasers, telescopes, and fast steerable mirrors:

Figure $$(1)$$: A schematic of the experimental apparatus taken from the team's paper. (Click to enlarge.)

The ground station for this experiment is an astronomical observatory located in Ngari, Tibet at an altitude of $$5100 \, \mathrm{m}$$, already well above much of Earth's atmosphere. The satellite receiver is called Micius, and its orbit is about $$500 \, \mathrm{km}$$ above Earth's surface. Micius' orbit is also what is called "sun-synchronous," meaning that the satellite passes over a given point on the Earth at the same time every day; in this case, midnight. Both the observatory and the satellite are equipped with reflecting telescopes with gimbaled mirrors, thus allowing the two telescopes to aim at each other as the satellite flies over Tibet. Additionally, the light passing through each telescope reflects off of one or more "faster-steering mirrors" (FSM), which allows for even more precise aiming. The aim is so precise, in fact, that the system can overcome the atmospheric turbulence that alters the path of light near Earth's surface! This turbulence is what causes stars to twinkle, and the twinkling of starlight was therefore observed and coupled into the system to help it correct for these atmospheric conditions. To lock on to each other, the observatory and the satellite exchanged laser pulses through their telescopes. Thus, the accuracy and precision needed to send and receive entangled photons was established.

Over the course of thirty-two midnights, for the three hundred and fifty seconds throughout which Micius was visible from Ngari, the apparatus diagramed in figure $$(1)$$ generated entangled photon pairs and shined one of them, photon $$3$$, at the satellite. The other photon, $$2$$, was then entangled with yet another photon, $$1$$, whose state was to be teleported. In each event, one of six different quantum states was teleported, combinations of the perpendicular linear photon polarization states; horizontal, $$\lvert H \rangle$$, and vertical, $$\lvert V \rangle$$ (basically, $$\vert 0 \rangle$$ and $$\vert 1 \rangle$$ for photons.):
\lvert \psi \rangle_1 = \begin{cases}
\lvert H \rangle_1 & \text{Horizontal Linear Polarization}, \\[2ex]
\lvert V \rangle_1 & \text{Vertical Linear Polarization}, \\[2ex]
\lvert + \rangle_1 = \frac{1}{\sqrt{2}} \left( \lvert H \rangle_1 + \lvert V \rangle_1 \right) & \text{Symmetric Superposition}, \\[2ex]
\lvert - \rangle_1 = \frac{1}{\sqrt{2}} \left( \lvert H \rangle_1 - \lvert V \rangle_1 \right) & \text{Antisymmetric Superposition}, \\[2ex]
\lvert R \rangle_1 = \frac{1}{\sqrt{2}} \left( \lvert H \rangle_1 + i \lvert V \rangle_1 \right) & \text{Right-Handed Circular Polarization}, \\[2ex]
\lvert L \rangle_1 = \frac{1}{\sqrt{2}} \left( \lvert H \rangle_1 - i \lvert V \rangle_1 \right) & \text{Left-Handed Circular Polarization}. \\[2ex]
\end{cases} \label{080417psi1}
Data collection was triggered by the detection of a fourth photon that was split off from photon $$1$$ prior to its entanglement with photon $$2$$. The entangled Bell state of photons $$1$$ and $$2$$ was then measured on the ground to commence teleportation while Micius caught photon $$3$$ and measured its state to see whether or not it matched the original state of photon $$1$$. The closeness of the match was quantified with a "fidelity" value that ranges between $$0$$ and $$1$$. It can be shown that in a world without quantum entanglement and teleportation, this value cannot be greater than $$2/3$$. Therefore, if a greater fidelity is achieved, then entanglement and teleportation must be in play. In all, data was captured for $$911$$ four-photon events, and the observered fidelities for each quantum state are shown in figure $$(2)$$:

Figure $$(2)$$: Experimental results from the team's paper. The dashed line represents the maximum fidelity without quantum entanglement and teleportation. The error bars represent one standard deviation calculated from the raw detection events. Fidelities are well above the $$2/3$$ limit for all quantum states.

Overall, an average fidelity of $$0.80 \pm 0.01$$ was obtained, which again is well above what one would expect in the absence of entanglement and teleportation. To quote the last line of the team's abstract,

Therefore, even though we are not going to have Star Trek style transporters anytime soon, and sensationalist headlines completely missed the point, we still have much about which to get excited!

So there you have it—live long and prosper, my friends.

🖖 Aaron

P.S. Click the "source" link below to read the original paper. It's interesting stuff, I promise!


text:
# Why Choose High Flow Microbial Air Samplers

Introduction

Most microbial air samplers sample at 100 litres per minute, taking 10 minutes to sample 1m3 of air. Ten minutes can be a long time waiting on site for something to complete; made even worse if multiple samples are needed with different media and in several zones within an area. This time represents a cost to your business, added downtime to production or disruption to your customers.

The use of high flow rate samplers may appear obvious in this case – shorter time to take samples. But, is this correct in real life situations and will a high flow rate sampler really be the best option?

This article presents a simple aid to calculating if there is such an advantage and presents two examples giving time saving calculations to demonstrate the benefits.

What are the time consuming factors when sampling?

The sampling time for a volume of air is not necessarily the key time consuming component of sampling activity. Many factors need to be considered including, but not limited to:

• Time to physically set-up the sampler
• Time to sterile wipe the sampling head and insert a media plate
• Time to operate the sampler controls to start sampling
• Time to unpack the equipment and pack away
• Size of the area to be tested and the number of sampling zones within that area
• The number of samples to be taken at each zone
• The number of media types being used at each zone

To gain a picture of aerosol contamination in a space, samples need to be taken at multiple locations to determine where the contamination is. Multiple samples at each of these points should be taken to produce a statistical mean of contamination levels – often requiring the use of two or more different media types to ascertain what type of organisms are present – bacteria or mould, etc.

Now, some mathematics

To help understand what benefit high flow samplers give, we developed a mathematical model to take key considerations into account, resulting in a total sampling time estimate.

T=n_z \left\lbrack n_p \;n_m \left(\frac{\frac{V_s }{U_f }+t_c +t_p }{n_s }+t_d \right)+t_s \right\rbrack    equation 1.

T Estimate of time to perform sampling

Vs Volume of air to be sampled per media plate, e.g. 1000 litres

Uf Sampler flow rate in litres per minute, eg. 100 or 180

nz Number of zones to sample

nm Types of media types to used

np Number of samples to be taken for each media type in each zone

ns Number of microbial air samplers being used

tc Time taken to change media plate

tp Time to programme/configure sampler to sample air

td Sampling start delay – i.e time to enable operator to leave area

ts Time to set-up a sampler in the required zone

All times are measured in minutes.

Limitations of the equation are:

• The same number of samples are taken at each zone within the space
• The same media types are used at each of those zones
• Will only work if:

Equation 1 is then used to calculate times T1 and T2 for both standard flow (Uf =100 lpm) and high flow (Uf =180 lpm) respectively, for the required sampling requirements.

The time saving found as:

T_s = T_1 - T_2 equation 2.

Alternatively, Equation 1 for both Uf cases can be substituted into equation 2 resulting in a time saving calculation that requires fewer variables to consider the saving:

T_s=\frac{n_m n_z n_p V_s(U_h-U_s)}{n_s U_h U_s} equation 3.

Uh High flow rate (e.g. 180 lpm)

Us Standard flow rate (e.g. 100 lpm)

To compare Cantium Scientific Limited MicroBio MB2 standard and high flow models, the time saving equation can be simplified further to:

T_s=\frac{n_m n_p n_z V_s}{225n_s} equation 4.

A simple cost saving estimate can be made by summing the hourly cost of lost production (Cp = $/hour) and the hourly rate of labour (CL =$/hour) involved in sampling, then multiplying by the saved time (Ts minutes):

S= \frac{C_p + C_L }{60 } \times T_s equation 5.

For all makes and models of microbial air samplers, combining equations 3 and 5 gives a cost saving equation of:

S= \frac{n_m n_z n_p V_s(U_h-U_s)(C_p + C_L) }{60 n_s U_h U_s } equation 6.

Example 1: Low volume single zone sampling

In this example, we consider the use of standard and high flow samplers. The requirement is to use a single media type and take three 200 litre samples. Firstly, we are assuming 1 minute to clean the head and insert a petri dish, 15 seconds to programme the sampler to take a sample, not use delayed start and take 3 minutes to physically set up a sampler, i.e. unpack and place on a tripod.

T_1=1 \times \left\lbrack 3 \times 1 \left(\frac{\frac{200 }{100 }+1 +0.25 }{1 }+0 \right)+3 \right\rbrack = 11.75 minutes

Using equation 1 for a 100 lpm (litre per minute) sampler, the total sampling time becomes:

Repeating the calculation for a 180 lpm sampler gives:

T_2=1 \times \left\lbrack 3 \times 1 \left(\frac{\frac{200 }{180 }+1 +0.25 }{1 }+0 \right)+3 \right\rbrack = 9.08 minutes

Saving = T_1 - T_2 = 2.67 minutes

In this example, a high flow sampler does not yield a great saving and their use would need careful consideration.

If production in a zone has to stop for a short period of time, the cost of lost production due to regular sampling would need to be considered and a high flow sampler may be a better option. If sampling can be undertaken while work continues, and sampling is only done occasionally, a standard 100 lpm flow rate sampler may be the economic option.

Example 2: Multiple zone sampling application

Here’s an example where a clean room requires sampling in 8 zones. Each zone will use two different media types with three samples on each of 1000 litres.

We are assuming 1 minute to clean the head and insert a petri dish, 15 seconds to programme the sampler to take a sample, 1 minute of delay for the operator to move away from the sampler and 3 minutes to physically set up each sampler, i.e. unpack and place on a tripod.

If two standard 100 lpm samplers are used, one for each media type, how long will sampling take?

By using equation 1 for the 100 lpm samplers, the time taken becomes:

T_1=8 \times \left\lbrack 3 \times 2 \left(\frac{\frac{1000 }{100 }+1 +0.25 }{2 }+1 \right)+3 \right\rbrack = 334 minutes

Now, if we use a 180 lpm samplers, the time to perform all samples is reduced to:

T_2=8 \times \left\lbrack 3 \times 2 \left(\frac{\frac{1000 }{180 }+1 +0.25 }{2 }+1 \right)+3 \right\rbrack = 227.3 minutes

Saving = T_1 - T_2 = 106.67 minutes

Saving over 1½ hours labour cost and reducing production interruption time to carry out sampling  will offer an enormous cost saving. In a very short time, the added cost of a high flow sampler will be recovered.

Equation 5 can now be used to determine that exact saving and payback time.Considerations when buying a high flow sampler?

If the decision has been made to purchase a high flow sampler, the following questions should be answered:

What’s the flow rate?

Most high flow devices will be in the range of 180 – 200 lpm. Very high flow rate samplers exist, but they are often large and require large vacuum pumps making them expensive and time consuming to set-up, negating any benefit of the faster flow rate. Be careful calculating savings with some samplers that describe high rates, such as 360 lpm. This may be achieved using two sampling heads running at 180 lpm each, so treat as two samplers for calculation purposes.

How big is it?

If the machine is big and heavy, it’ll take longer to move around and set-up. Ideally the machine should be easily handled in one hand and mountable on a tripod for the easiest use and quickest set-up time on site.

Is it easy to set-up and change sampling media?

What can slow operators down is the time taken to set-up the sampler, insert a petri dish or contact plate and then navigate through sometimes complex options to get it to take a sample.

Look for a machine with a sampling head that’s simple to clean between samples, is easy to insert and remove media plates from and has simple controls to start sampling with the fewest actions. Being easy to set-up on a tripod or site on a surface should also be considered.

How long does the battery last and is it easy to swap out in the field?

Many samplers on the market have built in battery packs that cannot be easily removed and swapped with a fully charged pack. With high flow machines enabling more samples to be taken in a day, the battery will more than likely run out before the day is over so consider looking for a machine where the battery can easily be swapped out and not having to take the sampler out of services while it is charged. Can a set of alkaline AA cells be used? Some also offer external DC power supplies so they can be mains powered and run without the worry of the battery fully discharging.

If a battery discharges during a sample, that sample will need to be retaken with a fresh plate, re-cleaning the sampling head and delaying other sampling work. This adds further cost and time.

How much does it cost?

The long term and operating costs should be considered. Do they use generic media in the form of contact plates or petri dishes that are readily available at low cost, or use proprietary media that may be more expensive with limited availability? Are spare parts available at low cost? If the built in rechargeable battery fails, and they will, can it be replaced at a sensible price?

Conclusion

Not every use case can justify the additional purchase cost of a high flow sampler. However, if you can answer yes to any of the following questions, then high flow models should be considered:

• Regular daily sampling with multiple plates
• Sampling multiple zones within an area
• Always sampling large volumes of air (>500m3) per sample
• Sampling with multiple media types

If performing many multiple samples, particularly when sampling requires the use of different media types, consideration should also be given to using a sampler for each of the media types or a multi-headed sampler.

To conclude, before choosing to buy a standard flow or high flow microbial sampler, perform a simple calculation using equations described in this paper to see how much time and money can be saved.

format: plain


text:
# Best class for making mount tougher in special gestalt 3.PF E6

Unfortunately, this question is going to require a lot of contextual explanation due to the strange, heavily tweaked system and classes in play. Bear with me.

### The system

I am playing in a game that uses a special, modified version of E6. The way it works is that we start with a “base” of being 6th-level commoners, and additional levels gained are gained using the “gestalt” rules from Unearthed Arcana. For example, a character might have started at our “1st” level as a commoner 6//fighter 1, and eventually progressed to commoner 6//fighter 6. Every six levels adds a new “layer” to the gestalt, so after commoner 6//fighter 6, one might become a commoner 6//fighter 6//bard 1. Each “layer” is not restricted to single-classing, so the following level could be commoner 6//fighter 6//bard 1/rogue 1 for example.

Other salient points:

• Feats are gained on even levels in each “layer” (including three from the commoner levels)
• Ability score boosts happen at each time we hit 3rd and 6th in a “layer.”
• All layers count towards prerequisites, including the commoner layer
• Commoner levels do not count towards plain “level” requirements or scaling, e.g. initiator level, the Leadership feat, etc.
• Prestige classes are on a case-by-case basis.
• All non-commoner classes are considered favored classes.

### The class

My particular character, a true-neutral halfling, is currently a commoner 6//convoker 4. Convoker is a utility/tank class that I am working on for Dreamscarred Press.1 Its origins were as an archetype for the spiritualist class called the goetic, which has a public playtesting document here. The convoker class has changed significantly since that playtest, unfortunately, so the precise details of the class are not available. The summary of the class remains the same, however: mage that transforms into eldritch horrors, and the eldritch horrors provide local control which allows the convoker to protect allies and manipulate his immediate environment.

Details on that:

• d10 HD
• Full BAB
• Good Will save
• 4+Int skills
• Simple weapons
• Light armor and shields (not tower shields)
• Close-range focus, with some features triggering on melee attacks
• 0-4th level occult (psychic) spells, in a progression that’s between bard and paladin (starts with 1st-level spells, but progresses slowly). Uses a small, fixed list of spells known (like the 3.5 beguiler, dread necromancer, or warmage).
• Numerous supernatural abilities gained by “splicing” one of the seven “horrors” to his body. These abilities are the ones most similar to what the goetic got.

### The mount, and the problem to solve

My character also has the Blood Steed trait, modified slightly to get him a riding dog rather than a horse. And this leads me to my question: I am playing a tanky character. My character has nearly 80 hp, very high for a character with 6 HD. My dog, that I ride into battle on, has just 13 hp, because she is a stock-standard riding dog. By the time I reach past commoner 6//convoker 6 and am looking at starting a new class, my character will have approximately 100 hp—and my dog will still have only 13.

The dog’s saves, particularly Will, are also large problems—I have been removed from a combat by her getting frightened and running away with my character on her, and would have been removed from a second fight if not for an ally’s timely remove fear.

What class or mix of classes, when gestalted with a commoner 6//convoker 6, will best beef up my riding dog and allow her to take similar levels of punishment as I do? Note that prestige classes enterable at 6th may be available, pending DM review, and alignment requirements are negotiable, but no more than 6 levels of any one class will ever be available. Combining my existing riding dog with a class-granted mount should be no problem. Classes that offer immediate results are preferred over classes that take time to reach the levels where improvements are offered.

### The campaign

Beyond the strange rules and the private class, some details about the character and campaign:

• The campaign is Age of Worms, so this is the Greyhawk cosmology.

• Str 14, Dex 8, Con 18, Int 14, Wis 19, Cha 8

• Psionic Body, Psionic Talent, Psionic Meditation, Blind-Fight, Psionic Weapon, Up the Walls, and Intuitive Fighting for current feats. There is also a convoker-specific feat that allows my mount to share in the movement-related abilities of the eldritch horrors I use. Note that Intuitive Fighting makes melee attack rolls Wisdom-based while psionically focused.

• Trained skills are Autohypnosis, Concentration, Handle Animal, Heal, Knowledge (arcana, dungeoneering, nature, planes, psionics, religion), Ride, and Spellcraft. Most have just one rank; Concentration, Knowledge (dungeoneering), and Spellcraft are maxed (6 ranks).

• The character is heavily influenced by Twoflower of Discworld fame: he is a foreign scholar on sabbatical to study the regions of the campaign, and he is exceptionally clueless on a variety of topics. Reflavoring classes is encouraged, so I don’t expect this to constrain choices much, but just in case.

• Despite high Wisdom, the character is easily distracted and often has his head in a book. I have elected not to roll Perception checks at all from time to time (and the dice have been “agreeable” in that I’ve often rolled poorly and failed Perception checks when I have rolled it). So a class that focuses heavily on being aware of one’s surroundings, having high Perception, and so on, might be inappropriate. Magical senses are quite acceptable, however; he already has a number, when he remembers to use them.

• The character venerates the Far Realms, and has little knowledge or contact with the regular Inner and Outer Planes. Heavily planar-themed classes, especially those that rely on summoning or calling creatures, would be inappropriate.

Again, reflavoring is OK, so for example, we could reflavor the celestial template offered by the paladin’s special mount to something appropriate, but something like malconvoker or thaumaturgist would be more problematic (and not just because they would do nothing to help my quandary).

• The convoker class has a lot of incentives to be in melee, several class features are based on melee attacks, and my character’s Dexterity is terrible. It would be very unlikely for him to switch to a ranged weapon.

• Both D&D 3.5e and Pathfinder material are acceptable, but please note sources. We are using Pathfinder skills (with some further merging beyond what Pathfinder does, and the return of Concentration as a skill), and CMB/CMD-based combat maneuvers. For classes and feats in both, we are allowed to use our choice of either, subject to some tweaking by the DM.

1. For the record, this weird, modified case is not being used as any kind of serious testing of the class; it’s just a game I am playing in with friends and got the OK to use the internal version of the class.
• Comments are not for extended discussion; this conversation has been moved to chat. – mxyzplk - SE stop being evil Jul 27 '17 at 12:18
• That moment when you read a huge Pathfinder question, foretaste a huge Pathfinder answer that KRyan could probably have already given, and then realize that KRyan actually asked that question... So sad. – Baskakov_Dmitriy Jul 27 '17 at 22:13
• @Baskakov_Dmitriy I do have some of my own thoughts, most of which I am seeing in answers, but I may yet add my own. I wanted to wait to see what others would suggest before doing that though. – KRyan Jul 27 '17 at 22:15

If you can meet the prerequisites of prestige classes by your previous gestalts, than it is reasonable to take Beastmaster as soon as possible. I assume you are going to finish your Convoker advancement, so the character would look like that:

Commoner 6 // Convoker 6 // Beastmaster 1

At the first level Beastmaster provides boost to your effective Druid level, so it will be a companion of 4 level Druid from the spot. I dont realy think you'll very much need any more levels in Beastmaster as it stacks with any companion advancing classes, and they have better class features.

From now on I see two options:

1. Take Druid for spellcastng, if you have room for it in your actions. Five levels will give you access to highest level spells even without that level lost due to Beastmaster. You also will have room for adding more prestige classes at those five levels later, as Druid is a regular class. Finaly, as a regular class, it has no prerequisites. It results in:

Commoner 6 // Convoker 6 // Beastmaster 1/Druid 5

2. Stack some other prestige and maybe regular classes with nice class features wich progresses your animal companion. I'd consider Wild Plains Outrider, good class with quick advancement and ability to make a full attack after mount's single move. Two remaining levels may be of practically any class wich progresses your animal companion, probably those of a Beastmaster or even some regular class whose class features you like. It results in:

Commoner 6 // Convoker 6 // Beastmaster 1/Wild Plains Outrider 3/xxx 2

I can't realy wrap my mind around Animal Ally feat mentioned by Chemus. I feel like, depending on your DM's judment, it may be quite usefull, but when to take it is a question I can't answer. Beastmaster and Wild Plains Outrider are both from Complete Adventurer.

• My character has finally reached that first 6th level, and I have taken Skill Focus (Handle Animal) in anticipation of beastmaster. Thanks. – KRyan Mar 30 '18 at 0:00

To add on to the above, if you can dip into 3.5, you might consider the Celestial Mount feat. Paladin lvl 4 and Exalted to get your mount some DR and some energy resistance. If you can dip further into 3.0, I'd suggest looking at the Windrider. One level of Windrider (prestige class) will give a hit die bonus to a chosen mount that should stack with anything. Of course, as a 3.0 prestige class, that one might be stretching things.

# Animal Companion feats/classes

In Pathfinder, the druid's animal companion mechanic is different from that of the 3.5 druid a bit. A major difference is that the companion has its own progression similar to that of a class.

Using the 3.5 animal companion is not strictly beneficial, as unlike a 3.5 Paladin's Special Mount, the Animal Companion does not share the character's base saving throws.

Using the Pathfinder druid's animal companion mechanics, I think that there are a few options that might suit, with varying levels of investment, and with varying results:

1. Since your variant of E6 still grants extra feats every other level, you could take the Animal Ally [General] feat, which grants an Animal Companion as a druid of $\text {your level }-3$, but requires the Nature Soul [General] feat as well.

Adding the Boon Companion [General] feat removes the 3 level penalty, giving you a 6th level animal companion (the maximum effective level with Boon companion is your character level).

This increases your Riding Dog's Hit dice, Base attack bonus, Strength and Dexterity bonus, Natural Armor bonus, known tricks, etc. It mitigates the Will save issue, via the +2 Base Will save bonus, as well as the Devotion ability.

2. The Beastmaster Prestige Class (Complete Adventurer, p. 26) requires 8 ranks (5 in Pathfinder) in Handle Animal (which you don't appear to have yet), 4 ranks (1 in Pathfinder) in Survival, and Skill Focus (Handle Animal) (again you'd have to be able to get it to qualify).

Its main benefit is that at 1st level, you gain a Companion as that of a druid of your Beastmaster level + 3. Thus, your riding dog would have abilities as 3 levels higher than your Beastmaster levels. So up to a 9th level animal companion. This class is used in many 'SuperMount' builds (See the next entry).

This actually stacks with the Animal Ally feat from above, though the feat must be taken before the class, as you're ineligible if you have an animal companion before taking the feat, to get you up to a 12th level animal companion (Boon companion does nothing for this, as your maximum effective druid level with it is your character level).

## This might be too much

1. While full Supermount may not really be possible in Pathfinder alone, since the Pathfinder version of Paladin has Divine Bond rather than Special Mount, you can get close without mixing the 3.5 paladin with PF ranger/druid, with possibly fairly extreme results. The core of the supermount build is having two (or more) sources of companion that another class then advances. Since, as mentioned in the dark wanderer's answer, the Hunter Hybrid class counts as both ranger and druid for advancing Animal Companion, this may stack doubly with the aforementioned Beastmaster, for $6\text{ (druid)}+(6-3)\text{ (ranger)}+(6+3)\text{ (beastmaster)}=18\text{th}$ level (perhaps 27th with Animal Ally) Animal Companion, though Pathfinder has only guidelines for advancement past level 20.

If you were to use 3.5 Paladin to power part of the Supermount, and the Pathfinder Hunter for the other, as above, then add in Halfling Outrider to finish it off, you could then add-in 6 level of paladin for special-mount and 6 virtual levels via Halfling Outrider (Complete Warrior p. 38) for the Special Mount of a 12th level Paladin (plus whatever other prestige classes you wanted to use to extend this further, such as Cavalier or Wild Plains Outrider), and the Animal Companion of a 39th level Druid (Halfling Outrider levels stack with paladin, and both druid and ranger levels for Mount/Companion levels).

Such a build could take much playing to produce. It could look like: Commoner 6//Convoker 6//Hunter 6//BeastMaster 6//Paladin 6//Halfling Outrider6.

• Both 3.5 and Pathfinder material is available. The 8 ranks requirement in 3.5 is equivalent to 5 in Pathfinder due to how PF skills work. And yes, feats still happen, on even levels. – KRyan Jul 27 '17 at 12:37
• @KRyan Ok, but I used both in my answer; I used Animal Ally rather than, say Wild Cohort, because the former already meshes with Pathfinder's rules, where the later has to be hammered to fit. Does 'both' mean that you can use both rulesets' versions of Base classes? – Chemus Jul 27 '17 at 13:17
• Yes, it does; there may be some negotiation on it, but both are potentially options. – KRyan Jul 27 '17 at 13:18
• @KRyan So the only bit that could be more powerful that I'm seeing is the supermount section. Are you wanting that level of optimization and investment? Or are you just asking me to fix my paladin reference? – Chemus Jul 27 '17 at 13:21
• "Class features that two classes share (such as uncanny dodge) accrue at the rate of the faster class". So, shouldn't Hunter 6 (effective Druid level 6) // Beastmaster 6 (effective Druid level 9) result in effective Druid level 9? – annoying imp Jul 27 '17 at 19:29

I think for your game the best class for this purpose is the Hunter. It's a hybrid druid/ranger class that focuses on making the animal companion really powerful. Notably, it's one of the only ways I'm aware of to get a companion with more effective druid levels than you have HD. At 6th level, you will have the companion of a 9th level druid, due to the Nature Training class ability that allows you to count your hunter levels as both druid and ranger levels (and thus add them together -3 for the purpose of your companion, if that's better than using your hunter levels straight).

This means your dog will have 8HD, a decent Con modifier, and Multiattack if you take this gestalt to 6th level. I don't know what you're doing about level up bonuses, but if you were a gnome you could possibly even give the dog DR6/magic. Since you're a halfling you have to settle for a +1 luck bonus on saving throws and 2 wasted levels, if that part of the system comes up. That's still better than a lot of the favored class options available, but it's not as good, and you probably picked commoner as your favored class anyways.

In addition to the 9 effective druid levels, the Hunter gets to apply one of a variety of effects to his companion, and can change the effect chosen as a swift action, via the animal focus feature. Notable defensive options include Evasion, +2 Con, +5 movement speed, +2 AC vs attacks of opportunity and +2 to attack rolls on attacks of opportunity, +4 Stealth, and +4 Perception.

Better still, the class punishes enemies who successfully kill your animal companion-- while your companion is dead you get the permanent and changeable buff you were previously giving it, and you also have the duration Summon Nature's Ally spells you cast go from 1 round per level to 1 minute per level, though you can only have one summon in effect at a time. If there's a cool convoker archetype that keys off Summon Nature's Ally instead of, idk, Summon Monster or Planar Binding or whatever it normally uses, that could be a pretty potent combo. At the very least, the potential punishment will help discourage enemies from attacking your animal companion with lethal force.

Furthermore, the class grants various bonuses to fighting in melee with your companion, like giving both you and your companion Outflank at level 2. While you'll need a third ally to help out, this can give you and your dog an extra +2 to hit in melee every now and then.

Plus your companion can retrain that feat once you hit level 3, since you then grant all teamwork feats you possess to your companion, though you also possess the ability to swap out your most recently gained teamwork feat for any other teamwork feat as a Standard Action (subject to Terms and Conditions) and that may well only be the Outflank feat you took for free at 2nd level, so maybe it's worth keeping (probably not), up to you.

4th, 5th, and 6th levels are less overwhelmingly awesome, but remember that these levels are when your hunter levels start counting for double for the purposes of animal companion power.

Now, given how much the Hunter gets in terms of their Animal Companion, you might be thinking that the rest of the class would be weak. Nope! You get Cleric BAB and better spells than a ranger, pulling from the "whatever's better" druid/ranger combo spell list and casting spontaneously because why not, starting at level 1. I mean, it's no druid, but it's very very solid.

The only weakness this class doesn't cover very well is, unfortunately, something you've already addressed: Will Saves, especially for Fear. Your companion will have amazing Fortitude and Reflex saves, but even with 9 effective druid levels, their Will save is going to be a measly 2+Wisdom. You can mitigate that significantly by having your companion take the Bodyguard archetype, granting it the ability to always act in the surprise round (always. It's a dire tortoise!) as well as access to the Heroic Defiance feat, which lets it delay stuff like fear effects for a round, which gives you time to decide what to do about it. You do lose Multiattack for Uncanny Dodge, though.

If you don't need the massive bonus of taking 6 levels of Hunter and are looking for more of a one level dip, Construct Rider lets you turn your dog into a robodog, granting it 20 bonus hit points and the amazing Construct Traits-- except their immunity to mind affecting effects, which is very disappointing. Construct rider doesn't play well with Hunter in terms of hit points, because Hunter gives you tons of Hit Dice and a huge Con mod if you want it, and Construct Rider replaces the Con mod with a static bonus that isn't quite as good by 6th level. If you care less about hit points and more about Construct Traits being awesome, it could still be a good trade, but you aren't getting anything else from this level (Alchemist 1 isn't really good for anything for you, and you don't even get that, and the druid animal companion progression isn't as good as the hunter progression so they don't stack) so it's not worth it.

In concnlusion, Hunter 6 is incredibly powerful in terms of boosting your animal companion's survivability in an E6 game, and Construct Rider 1 is a maybe-better 1 level dip.

P.S. If you are playing with the pre-errata version of the Primal Hunter archetype that is hands down the best choice, but it's stupid and you shouldn't do that. It trades that versatile +2 to Con thing for full summoner eidolon progression on your animal companion. Like, it's better-than-full animal companion, plus better-than-ranger spellcasting, plus everything the summoner gets except spellcasting stacked on top of that better-than-full animal companion chassis, and if the enemy manages to kill it it just transfers all that to you. It's ridiculous to the point that they errataed it, and now it only gives you 2 evolution points for a handful of minutes a day, though they are now mutable instead of set for each level.

• Comments are not for extended discussion; this conversation has been moved to chat. – mxyzplk - SE stop being evil Jul 27 '17 at 12:18
• As the chat room is unavailable (at least from an above link) I think it is convenient to link a follow up question, discussing сontroversial issue of this answer (and probable reason of a downvote). – annoying imp Aug 11 '17 at 6:12

Well, paladin might be good, because of the Heal Mount spell, but I'm a noob, so there might be better classes. Maybe Druid 4/Commoner 6/ Cleric 4/ Hunter 4(3 animal Companions!)?

• I don’t want to be too harsh to a new user, but there are a lot of problems with this answer. We expect answers to show a bit of expertise here; we want to emphasize answers with valuable information to add, so adding “I'm a noob” doesn’t really help your case. Moreover, the question is pretty detailed about how the system here works, and among other things, heal mount is impossible to achieve, being a 3rd-level paladin spell and the level cap being 6th. And the gestalt rules don’t work neatly with repeat class features (e.g. three animal companions). I appreciate you trying to help, but... – KRyan Jul 31 '17 at 15:27


text:
## Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression

Aymeric Dieuleveut, Nicolas Flammarion, Francis Bach; 18(101):1−51, 2017.

### Abstract

We consider the optimization of a quadratic objective function whose gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a zero-mean finite variance random error. We present the first algorithm that achieves jointly the optimal prediction error rates for least-squares regression, both in terms of forgetting the initial conditions in $O(1/n^2)$, and in terms of dependence on the noise and dimension $d$ of the problem, as $O(d/n)$. Our new algorithm is based on averaged accelerated regularized gradient descent, and may also be analyzed through finer assumptions on initial conditions and the Hessian matrix, leading to dimension- free quantities that may still be small in some distances while the “optimal” terms above are large. In order to characterize the tightness of these new bounds, we consider an application to non-parametric regression and use the known lower bounds on the statistical performance (without computational limits), which happen to match our bounds obtained from a single pass on the data and thus show optimality of our algorithm in a wide variety of particular trade-offs between bias and variance.

[abs][pdf][bib]


text:
Tutorial 30: MediaWiki Page History¶

Here, we return to our study of the MediaWiki API. This will be a useful review of how to grab and parse data, both are key steps in doing work in data science. Specifically, we will see how to grab and parse the history of a wikipedia page.

In [1]:
import json
import os
from os.path import join
import re
import requests

import wiki
import iplot
import matplotlib.pyplot as plt

I am going to demonstrate at first using the Wikipedia page for coffee. You'll be able to look at other pages later in the tutorial. To start, let's grab the page JSON file (either from disk or downloaded through the functions provided in wiki.py). I'll also print out the keys in the returned dictionary object.

In [2]:
page_json = wiki.get_wiki_json("Coffee")
page_json.keys()

Out[2]:
dict_keys(['title', 'pageid', 'revid', 'redirects', 'text', 'langlinks', 'categories', 'links', 'templates', 'images', 'externallinks', 'sections', 'parsewarnings', 'displaytitle', 'iwlinks', 'properties'])

We have not done anything with the 'revid', but this will be key for our work today. The 'revid' is a unique key that describes a particular version of a page. It is unique on it's own; you do not need to combine it with the 'pageid' (I found this confusing in the MediaWiki documentation).

Query API¶

In order to get previous revision ids for a given page, we need to use the 'query' API. Last time, and throughout the semester so far, we have been only using the 'parse' functionality of the MediaWiki API. Let's see how the query API works. Recall that we need a base API URL that the query is sent to. This remains unchanged from tutorial 8:

In [3]:
lang = 'en'
base_api_url = 'https://' + lang + '.wikipedia.org/w/api.php?'
base_api_url

Out[3]:
'https://en.wikipedia.org/w/api.php?'

Now, we produce a query by providing variable keys separated by the amperstand (&) symbol. For the query API we need to specify the following:

• action=query to describe the action that we want to take
• format=json to let the API know we want data returned as JSON objects
• prop=revisions to tell the API that we want to see the page revisions
• rvprop=ids|flags|timestamp|comment|user indicate what data to return about the revision
• rvstartid=###### to indicate the revision we want to start at, with the proper revid filled in for the ########

Start by extracting the revid for the Coffee page:

In [4]:
pageid = page_json['pageid']
pageid

Out[4]:
604727
In [5]:
revid = page_json['revid']
revid

Out[5]:
865835930

And then construct the API query:

In [6]:
api_query = base_api_url + "action=query&" + "format=json&" + \
"prop=revisions&" + "rvprop=ids|flags|timestamp|comment|user&" \
"pageids={0:d}&".format(pageid) + \
"rvstartid={0:d}&".format(revid)
api_query

Out[6]:
'https://en.wikipedia.org/w/api.php?action=query&format=json&prop=revisions&rvprop=ids|flags|timestamp|comment|user&pageids=604727&rvstartid=865835930&'

Finally, we can call the API and have the page data returned to us. Take a few minutes to look at the output before moving on the next section.

In [7]:
r = requests.get(api_query)
page_data = r.json()
page_data

Out[7]:
{'continue': {'rvcontinue': '20181003154447|862314871', 'continue': '||'},
'query': {'pages': {'604727': {'pageid': 604727,
'ns': 0,
'title': 'Coffee',
'revisions': [{'revid': 865835930,
'parentid': 865835769,
'timestamp': '2018-10-26T13:55:56Z',
'comment': '/* Coffeehouses */'},
{'revid': 865835769,
'parentid': 865834651,
'timestamp': '2018-10-26T13:54:31Z',
'comment': '/* Coffeehouses */'},
{'revid': 865834651,
'parentid': 864501853,
'minor': '',
'timestamp': '2018-10-26T13:44:33Z',
'comment': 'no need for those pictures in the lead'},
{'revid': 864501853,
'parentid': 864501313,
'user': 'Heroeswithmetaphors',
'timestamp': '2018-10-17T17:07:50Z',
{'revid': 864501313,
'parentid': 864501192,
'user': 'Heroeswithmetaphors',
'timestamp': '2018-10-17T17:03:25Z',
{'revid': 864501192,
'parentid': 863262382,
'user': 'Heroeswithmetaphors',
'timestamp': '2018-10-17T17:02:20Z',
{'revid': 863262382,
'parentid': 863213758,
'user': 'Funfactsandwheretofindthem',
'timestamp': '2018-10-09T18:06:40Z',
'comment': "/* Coffeehouses */ Updated Starbucks' outlet numbers from 2010 date"},
{'revid': 863213758,
'parentid': 863088071,
'minor': '',
'timestamp': '2018-10-09T12:08:40Z',
'comment': ''},
{'revid': 863088071,
'parentid': 863082281,
'user': 'Zefr',
'timestamp': '2018-10-08T17:00:11Z',
'comment': '/* Method of action */ add more focused review using [[WP:UCB|Citation bot]]'},
{'revid': 863082281,
'parentid': 862314871,
'user': 'Supersarah1121',
'timestamp': '2018-10-08T16:13:31Z',
'comment': 'Medical Citation Added'}]}}}}

What information is shown about revision? What information does the parameter 'parentid' give you about the revision? How many revisions are returned here?

Parsing page dates¶

I want you to try writing some code again. Cycle through the output of page_data to produce a python list named rev_years that gives the year of each revision. Note: you might start by storing the entire time stamp and then modify the code to store just the year. Second note: You don't need to be too clever to get the year from the time stamp, just take the first four characters in the string.

In [8]:
revisions = page_data['query']['pages']['604727']['revisions']
len(revisions)

Out[8]:
10
In [9]:
rev_years = []

for rev in revisions:
rev_years.append(rev['timestamp'][:4])

rev_years

Out[9]:
['2018',
'2018',
'2018',
'2018',
'2018',
'2018',
'2018',
'2018',
'2018',
'2018']

Once you have created the list rev_years, run the following to show how many revisions there are in each year.

In [10]:
import collections

collections.Counter(rev_years)

Out[10]:
Counter({'2018': 10})

The output likely won't be too interesting yet because you only have the first 10 revisions of the page. Let's see how to rectify that now.

Increasing the page limit¶

By default, MediaWiki returns only the last 10 revisions. We can fetch up to 500 revisions by adding the parameter rvlimit=max to the query. Modify the variable api_query and fetch the modifyied data. I suggest NOT printing out the results of page_data because there will be a lot of them!

In [11]:
api_query = base_api_url + "action=query&" + "format=json&" + \
"prop=revisions&" + "rvprop=ids|flags|timestamp|comment|user&" \
"rvlimit=max&" + \
"pageids={0:d}&".format(pageid) + \
"rvstartid={0:d}&".format(revid)
r = requests.get(api_query)
page_data = r.json()

revisions = page_data['query']['pages']['604727']['revisions']

Now, copy the code you had above to find the distribution of years for the first 500 revisions.

In [12]:
rev_years = []

for rev in revisions:
rev_years.append(rev['timestamp'][:4])

collections.Counter(rev_years)

Out[12]:
Counter({'2018': 112, '2017': 386, '2016': 2})

You should see about 112 revisions for 2018, 386 for 2017, and just 2 for 2016. This is a bit more interesting, but still not the whole story because even the maximum query returns just 500 pages.

Continuing the query¶

In order to get more results from the API we must place another query request. This is similar to seeing the "next page" of results on a site such as Google or Amazon when searching for a webpage or product. The idea here is common to many API's such as the ones used by Google, Twitter, and Facebook.

Notice that the variable page_data contains an element named continue:

In [13]:
page_data['continue']

Out[13]:
{'rvcontinue': '20161214161444|754802248', 'continue': '||'}

Create a variable api_query_continue that adds the parameter rvcontinue=###### with the ##### filled in from the above value to the variable api_query.

In [14]:
api_query_continue = api_query + "rvcontinue=" + page_data['continue']['rvcontinue']

Next, call this query and load the data into the variable page_data.

In [15]:
r = requests.get(api_query_continue)
page_data = r.json()

And now see the distribution of page years in this new chunk of data:

In [16]:
rev_years = []

for rev in page_data['query']['pages'][str(pageid)]['revisions']:
rev_years.append(rev['timestamp'][:4])

collections.Counter(rev_years)

Out[16]:
Counter({'2016': 322, '2015': 178})

You should see about 322 revisions in 2016 and another 178 in 2015.

Getting all of the pages¶

We still do not have all of the Coffee pages, just the next 500 of them. In order to get all of the pages we have to cycle through these continue statements until we reach the end of the pages. Let's write the code to handle this now. Rather than just grabbing the years, we will construct a list of all the information about each revision.

I've written the function here for you to generate the list of revisions, but you should be able to understand what is going on in all of the code. It spits out the progress of the API by showing the number of revisions grabbed as well as indicating what the last timestamp grabbed was.

In [17]:
def wiki_page_revisions(page_title):
page_json = wiki.get_wiki_json(page_title)
pageid = page_json['pageid']
revid = page_json['revid']

api_query = base_api_url + "action=query&" + "format=json&" + \
"prop=revisions&" + "rvprop=ids|flags|timestamp|comment|user&" \
"rvlimit=max&" + \
"pageids={0:d}&".format(pageid) + \
"rvstartid={0:d}&".format(revid)
r = requests.get(api_query)
page_data = r.json()

rev_data = page_data['query']['pages'][str(pageid)]['revisions']

while 'continue' in page_data:
api_query_continue = api_query + \
"rvcontinue={0:s}&".format(page_data['continue']['rvcontinue'])
r = requests.get(api_query_continue)
page_data = r.json()
rev_data += page_data['query']['pages'][str(pageid)]['revisions']
msg = "Loaded {0:d} revisions, through {1:s}"
print(msg.format(len(rev_data), rev_data[-1]['timestamp']))

return rev_data

In [18]:
rev_data = wiki_page_revisions("Coffee")

Loaded 1000 revisions, through 2015-10-06T08:23:46Z

Just looking at the message output, how many revisions have been made to the Coffee page and when was the page first created?

Modify your code you used above to grab the list rev_years from the list rev_data.

In [19]:
 rev_years = []

for rev in rev_data:
rev_years.append(rev['timestamp'][:4])

collections.Counter(rev_years)

Out[19]:
Counter({'2018': 112,
'2017': 386,
'2016': 324,
'2015': 533,
'2014': 532,
'2013': 682,
'2012': 589,
'2011': 624,
'2010': 1065,
'2009': 1059,
'2008': 1073,
'2007': 1842,
'2006': 1594,
'2005': 692,
'2004': 129})

In what year were the most revisions completed? Has 2018 had an unusually high or lower number of revisions at this point in the year?

Finally, you can even use the following code to produce a line plot of the number of revisions in each year.

In [20]:
plt.rcParams["figure.figsize"] = (12, 10)

In [21]:
cnt = collections.Counter(rev_years).items()
cnt = sorted(cnt, key=lambda x: x[0])
plt.xticks(rotation=90)
plt.plot([x[0] for x in cnt], [x[1] for x in cnt], 'k-', lw=2)

Out[21]:
[<matplotlib.lines.Line2D at 0x11fbbb3c8>]

Revision data¶

We now have some information about each of the page revisions, but we still have not seen how to grab the actual page data from a given revision. To do this, we need to return to the "parse" API with our revision ids in hand. Essentially, all we need to do is call the parse action, specify the format as JSON, and provid the revid to the parameter oldid. So, to get the very first version of the coffee page we would first get the revision id:

In [22]:
revid = rev_data[-1]['revid']
revid

Out[22]:
3245467

And then place the following API query:

In [23]:
api_query = base_api_url + "action=parse&" + "format=json&" + \
"oldid={0:d}&".format(revid)
r = requests.get(api_query)
page_data = r.json()['parse']

This page is now in the format we have been working with the rest of the semester but gives and old version of the page, way back from 2014.

In [24]:
page_data.keys()

Out[24]:
dict_keys(['title', 'pageid', 'revid', 'text', 'langlinks', 'categories', 'links', 'templates', 'images', 'externallinks', 'sections', 'parsewarnings', 'displaytitle', 'iwlinks', 'properties'])

It would be fun to see what this page actually looks like rendered as html. Let's write it to disk with some header information to make it look reasonable:

In [25]:
with open('temp.html', 'w', encoding="UTF-8") as fin:
fin.writelines("")
fin.writelines(page_data['text']['*'])
fin.writelines("")

You should see that the page was very basic back in 2004!

A different page¶

Repeat the code above (you can drop all of the steps into one or two code blocks) to look at another page that interests you.

In [27]:
rev_data = wiki_page_revisions("Data Science")
rev_years = []

for rev in rev_data:
rev_years.append(rev['timestamp'][:4])

cnt = collections.Counter(rev_years).items()
cnt = sorted(cnt, key=lambda x: x[0])
plt.xticks(rotation=90)
plt.plot([x[0] for x in cnt], [x[1] for x in cnt], 'k-', lw=2)

Pulling data from MediaWiki API: 'Data Science'

Out[27]:
[<matplotlib.lines.Line2D at 0x11e13ab00>]

How does the pattern of number of changes differ from the coffee page?

What next¶

We certainly could grab the revision history for every change that has been made to the Coffee page. Most changes, though, are not particularly interesting on their own (at least for our level of study here). Instead, I want to focus on large-scale change over time by grabbing one page for each year in the collection. Starting with the values in rev_data, I want you to use the space below to grab the last version of the pages on Coffee for each year. That is, start with the current page, then get the last page from 2017, then 2016, and so forth until you get back to the page at the end of 2004. Store these pages as a list named page_history.

In [28]:
# Note: run this cell just once to refresh the value in rev_data
# from running the code to grab a different page above
rev_data = wiki_page_revisions("Coffee")

Loaded 1000 revisions, through 2015-10-06T08:23:46Z

In [49]:
base_api_url = 'https://' + lang + '.wikipedia.org/w/api.php?'

page_history = []
last_year = int(rev_data[0]['timestamp'][:4]) + 1

for rev in rev_data:
this_year = int(rev['timestamp'][:4])
if this_year < last_year:
last_year = this_year

# grab the page
revid = rev['revid']
api_query = base_api_url + "action=parse&" + "format=json&" + \
"oldid={0:d}&".format(revid)
r = requests.get(api_query)
page_data = r.json()['parse']

page_history.append((rev, page_data))

# output progress
print("Grabbed page at {0:d}".format(revid))

Grabbed page at 865835930
Grabbed page at 817267881
Grabbed page at 755468653
Grabbed page at 697272803
Grabbed page at 640191769
Grabbed page at 588559107
Grabbed page at 530615673
Grabbed page at 468752501
Grabbed page at 405119400
Grabbed page at 335170993
Grabbed page at 260942162
Grabbed page at 181074859
Grabbed page at 97628123
Grabbed page at 33382948
Grabbed page at 9036398

Then, when you are done with that, cycle through the pages to extract the length of the page in characters, the number of internal links, the number of external links, the number of images, and the number of sections.

In [59]:
import xml.etree.ElementTree as ET
import pandas as pd

meta = dict(timestamp=[], pageid=[], revid=[], link=[], title=[], year=[],

for rev, page_data in page_history:
tree = ET.fromstring(page_data['text']['*'])
meta['timestamp'].append(rev['timestamp'])
meta['pageid'].append(page_data['pageid'])
meta['revid'].append(rev['revid'])
meta['title'].append(re.sub('<[^>]+>', '', page_data['displaytitle']))
meta['year'].append(int(rev['timestamp'][:4]))
meta['num_chars'].append(len(page_data['text']['*']))
meta['num_p'].append(len(tree.findall(".//p")))
meta['num_sections'].append(len(page_data['sections']))
meta['num_images'].append(len(page_data['images']))

pdf = pd.DataFrame(meta).drop_duplicates()
pdf

Out[59]:
0 2018-10-26T13:55:56Z 604727 865835930 Coffee Coffee 2018 448075 130 42 43 681 241 169
1 2017-12-27T07:58:07Z 604727 817267881 Coffee Coffee 2017 437723 128 40 47 660 231 169
2 2016-12-18T05:42:22Z 604727 755468653 Coffee Coffee 2016 459558 133 44 50 652 239 169
3 2015-12-29T11:13:18Z 604727 697272803 Coffee Coffee 2015 433789 126 42 44 644 198 169
4 2014-12-30T04:41:08Z 604727 640191769 Coffee Coffee 2014 421406 126 44 58 595 183 169
5 2013-12-31T18:54:37Z 604727 588559107 Coffee Coffee 2013 374642 129 43 56 556 140 169
6 2012-12-31T14:43:47Z 604727 530615673 Coffee Coffee 2012 360744 119 38 48 537 152 169
7 2011-12-31T10:26:36Z 604727 468752501 Coffee Coffee 2011 358617 114 33 54 536 151 169
8 2010-12-31T05:22:50Z 604727 405119400 Coffee Coffee 2010 328394 89 32 49 498 139 170
9 2009-12-31T20:57:40Z 604727 335170993 Coffee Coffee 2009 244457 65 23 46 391 103 170
10 2008-12-30T20:11:56Z 604727 260942162 Coffee Coffee 2008 194285 50 21 35 320 89 170
11 2007-12-30T23:01:42Z 604727 181074859 Coffee Coffee 2007 160272 39 16 18 299 68 169
12 2006-12-31T23:39:41Z 604727 97628123 Coffee Coffee 2006 95116 31 21 14 234 25 169
13 2005-12-31T15:36:40Z 604727 33382948 Coffee Coffee 2005 114129 79 39 18 298 34 169
14 2004-12-30T20:13:34Z 604727 9036398 Coffee Coffee 2004 33909 33 15 3 86 5 169

Finally, plot the data across time for each of these variables. For example, if you stored the data for the number of internal links as a list named num_ilinks, you should be able to run something like this:

In [60]:
plt.xticks(rotation=90)
plt.plot(list(range(2018,2003,-1)), pdf['num_chars'], 'k-', lw=2)

Out[60]:
[<matplotlib.lines.Line2D at 0x11da4c048>]
In [61]:
plt.xticks(rotation=90)
plt.plot(list(range(2018,2003,-1)), pdf['num_p'], 'k-', lw=2)

Out[61]:
[<matplotlib.lines.Line2D at 0x120e34eb8>]
In [62]:
plt.xticks(rotation=90)
plt.plot(list(range(2018,2003,-1)), pdf['num_sections'], 'k-', lw=2)

Out[62]:
[<matplotlib.lines.Line2D at 0x12064ec50>]
In [63]:
plt.xticks(rotation=90)
plt.plot(list(range(2018,2003,-1)), pdf['num_images'], 'k-', lw=2)

Out[63]:
[<matplotlib.lines.Line2D at 0x121452e10>]
In [64]:
plt.xticks(rotation=90)

Out[64]:
[<matplotlib.lines.Line2D at 0x1206e5710>]
In [66]:
plt.xticks(rotation=90)
plt.plot(list(range(2018,2003,-1)), pdf['num_langs'], 'k-', lw=2)

Out[66]:
[<matplotlib.lines.Line2D at 0x1206e07f0>]
In [ ]:
plt.xticks(rotation=90)

Take note of any interesting patterns that arise over time with the pages.

Even more practice¶

My guess is that the above tasks will take up most of the class time. If you want extra practice or finish early, wrap up the code above in a function that takes just a page name and returns all of the metrics as a panda's DataFrame object. Include the timestamp as the first column of the data frame and make sure that you handle the case where the number of years may be different and there may even be no revisions in a given year.

In [68]:
def get_page_history(page_title):

rev_data = wiki_page_revisions(page_title)
base_api_url = 'https://' + lang + '.wikipedia.org/w/api.php?'

page_history = []
last_year = int(rev_data[0]['timestamp'][:4]) + 1

for rev in rev_data:
this_year = int(rev['timestamp'][:4])
if this_year < last_year:
last_year = this_year

# grab the page
revid = rev['revid']
api_query = base_api_url + "action=parse&" + "format=json&" + \
"oldid={0:d}&".format(revid)
r = requests.get(api_query)
page_data = r.json()['parse']

page_history.append((rev, page_data))

# output progress
print("Grabbed page at {0:d}".format(revid))

return page_history

In [69]:
def get_page_history_meta(page_title):
import xml.etree.ElementTree as ET
import pandas as pd

page_history = get_page_history(page_title)

meta = dict(timestamp=[], pageid=[], revid=[], link=[], title=[], year=[],

for rev, page_data in page_history:
tree = ET.fromstring(page_data['text']['*'])
meta['timestamp'].append(rev['timestamp'])
meta['pageid'].append(page_data['pageid'])
meta['revid'].append(rev['revid'])
meta['title'].append(re.sub('<[^>]+>', '', page_data['displaytitle']))
meta['year'].append(int(rev['timestamp'][:4]))
meta['num_chars'].append(len(page_data['text']['*']))
meta['num_p'].append(len(tree.findall(".//p")))
meta['num_sections'].append(len(page_data['sections']))
meta['num_images'].append(len(page_data['images']))

pdf = pd.DataFrame(meta).drop_duplicates()
return pdf

In [71]:
pdf = get_page_history_meta('Statistics')
pdf

Loaded 1000 revisions, through 2013-04-30T23:42:38Z
Grabbed page at 867661767
Grabbed page at 817612155
Grabbed page at 754986487
Grabbed page at 697433595
Grabbed page at 639300278
Grabbed page at 588463335
Grabbed page at 530307227
Grabbed page at 468573297
Grabbed page at 403631349
Grabbed page at 334749989
Grabbed page at 260637347
Grabbed page at 181279972
Grabbed page at 97370751
Grabbed page at 33018695
Grabbed page at 9191083
Grabbed page at 2220137
Grabbed page at 556462
Grabbed page at 279816

Out[71]:
0 2018-11-07T05:14:26Z 26685 867661767 Statistics Statistics 2018 240263 69 30 23 642 54 131
1 2017-12-29T14:02:01Z 26685 817612155 Statistics Statistics 2017 217251 71 30 16 575 47 131
2 2016-12-15T16:32:00Z 26685 754986487 Statistics Statistics 2016 220733 69 30 25 578 43 131
3 2015-12-30T14:08:50Z 26685 697433595 Statistics Statistics 2015 213930 68 30 24 574 38 131
4 2014-12-23T07:13:33Z 26685 639300278 Statistics Statistics 2014 191070 67 28 24 547 17 131
5 2013-12-31T02:50:14Z 26685 588463335 Statistics Statistics 2013 165371 53 20 18 528 20 131
6 2012-12-29T15:29:36Z 26685 530307227 Statistics Statistics 2012 153761 49 20 15 514 12 131
7 2011-12-30T10:17:04Z 26685 468573297 Statistics Statistics 2011 149561 47 24 14 506 22 131
8 2010-12-22T01:49:03Z 26685 403631349 Statistics Statistics 2010 145910 46 24 15 501 22 131
9 2009-12-29T17:08:16Z 26685 334749989 Statistics Statistics 2009 140638 49 24 14 490 23 131
10 2008-12-29T07:58:39Z 26685 260637347 Statistics Statistics 2008 124992 36 16 15 488 13 131
11 2007-12-31T22:52:04Z 26685 181279972 Statistics Statistics 2007 137766 49 20 9 539 39 131
12 2006-12-30T18:06:41Z 26685 97370751 Statistics Statistics 2006 68782 47 21 3 214 27 131
13 2005-12-28T18:16:20Z 26685 33018695 Statistics Statistics 2005 52753 25 17 6 182 45 131
14 2004-12-31T01:51:03Z 26685 9191083 Statistics Statistics 2004 19079 18 8 0 69 18 131
15 2003-12-31T18:44:55Z 26685 2220137 Statistics Statistics 2003 9822 12 3 0 53 3 131
16 2002-12-28T08:47:09Z 26685 556462 Statistics Statistics 2002 2743 4 2 0 12 1 131
17 2001-11-03T02:49:39Z 26685 279816 Statistics Statistics 2001 3055 12 0 0 10 0 131


text:
In a lockable obfuscation scheme a party takes as input a program $P$, a lock value $\alpha$, a message $m$ and produces an obfuscated program $\tilde{P}$. The obfuscated program can be evaluated on an input $x$ to learn the message $m$ if $P(x)= \alpha$. The security of such schemes states that if $\alpha$ is randomly chosen (independent of $P$ and $m$), then one cannot distinguish an obfuscation of $P$ from a “dummy'' obfuscation.
Existing constructions of lockable obfuscation achieve provable security under the Learning with Errors assumption. One limitation of these constructions is that they achieve only statistical correctness and allow for a possible one sided error where the obfuscated program could output the $m$ on some value $x$ where $P(x) \neq \alpha$. In this work we motivate the problem of studying perfect correctness in lockable obfuscation for the case where the party performing the obfuscation might wish to inject a backdoor or hole in correctness. We begin by studying the existing constructions and identify two components that are susceptible to imperfect correctness. The first is in the LWE-based pseudo random generators (PRGs) that are non-injective, while the second is in the last level testing procedure of the core constructions. We address each in turn. First, we build upon previous work to design injective PRGs that are provably secure from the LWE assumption. Next, we design an alternative last level testing procedure that has additional structure to prevent correctness errors. We then provide a surgical proof of security (to avoid redundancy) that connects our construction to the construction by Goyal, Koppula, and Waters (GKW). Specifically, we show how for a random value $\alpha$ an obfuscation under our new construction is indistinguishable from an obfuscation under the existing GKW construction.

All News


text:
Artisan Vegan Cheese Ireland, Do Not Go Gentle Into That Good Night Tone, Rc Ii-6 Self Concept Examples, School Improvement Strategies, Glymed Plus Skin Astringent 5, Felidar Retreat Scg, Green Tea Water Bomb Bonajour, Pangasius Fish Seed In Andhra Pradesh, Badgercare Eligibility Handbook, What Do Dog Contractions Look Like, How Is Rain, Steam And Speed Visually Constructed, " /> Artisan Vegan Cheese Ireland, Do Not Go Gentle Into That Good Night Tone, Rc Ii-6 Self Concept Examples, School Improvement Strategies, Glymed Plus Skin Astringent 5, Felidar Retreat Scg, Green Tea Water Bomb Bonajour, Pangasius Fish Seed In Andhra Pradesh, Badgercare Eligibility Handbook, What Do Dog Contractions Look Like, How Is Rain, Steam And Speed Visually Constructed, " />

# quality management in software industry

###### Deluxe Red Door Panel
March 29, 2020

All arithmetic operations can be performed on the resulting count. It proposes using 4 elements: quality management system, function of quality control, planning quality and quality controls. The different external attributes are productivity, experience, quality, usability, reliability, comfort etc. It is also based on the CBA IPI. At this level, the measurement of requirement characteristics is difficult. Basically, a three-level structure of management exists in software development organizations −, Following are the responsibilities of the top management in ensuring Software Quality −, Assure the quality of the company’s software products and software maintenance services, Communicate the importance of the product and service quality in addition to customer satisfaction to employees at all levels, Assure satisfactory functioning and full compliance with customer requirements, Ensure that quality objectives are established for the organization’s SQA system and that its objectives are accomplished, Initiate planning and oversee implementation of changes necessary to adapt the SQA system to major internal as well as external changes related to the organization’s clientele, competition, and technology, Intervene directly to support resolution of crisis situations and minimize damages, Ensure the availability of resources required by SQA systems, The following steps can be taken by the top management to fulfill its responsibilities −. Models are useful for interpreting the behavior of the numerical elements of the real-world entities as well as measuring them. Most of the time, it takes several months between the tender submission and the signing of the contract. It presents a particularly difficult problem when designing an experiment or case study, because their predictions often affect the outcome. The hypothesis is the tentative theory or supposition that the programmer thinks explains the behavior they want to explore. In case of software, product production planning and manufacturing phases are not required as the manufacturing of software copies and the printing of software manuals are conducted automatically. Measurement of structural properties of a software is important for estimating the development effort as well as for the maintenance of the product. Some stochastic prediction systems are more stochastic than others. This metric is a key concept of the defect removal model for software development. The metric of percent defective fixes is the percentage of all fixes in a time interval that is defective. Research and publish the best content. The basic metrics for these tokens were, The volume of program = No. With quality management standards, organizations can steadily assure that their software products achieve an acceptable level of quality. The effort required to generate P is given by, $$E = V\diagup L = \frac{\mu_{1}N_{2}Nlog_{2}\mu}{2\mu_{2}}$$, Where the unit of measurement E is elementary mental discriminations needed to understand P, The other alternatives for measuring the length are −, In terms of the number of bytes of computer storage required for the program text, In terms of the number of characters in the program text. For example, maintainability but each of the factors is to be measured to check for the content and the quality control. The numbers represent ranking only. OGC (Office of Government Commerce) (2009). The input measures can be the size and volatility of the requirements. To perform the mapping, we have to specify domain, range as well as the rules to perform the mapping. The attribute always takes the form “number of occurrences of x in the entity”. The GQM paradigm, in concert with the process maturity, has been used as the basis for several tools that assist managers in designing measurement programs. External attributes are those that can be measured only with respect to its relation with the environment. A defective fix can be recorded in two ways: Record it in the month it was discovered or record it in the month the fix was delivered. If the number of defects is large, then the small value of the percentage metric will show an optimistic picture. Are they appropriately precise? Usually the longer the latency, the more will be the customers that get affected. In this case, we have no control over the situation at hand. These factors are as follows −. Examples include the number of software developers, the staffing pattern over the life cycle of the software, cost, schedule, and productivity. This assures that the project commitments have been clearly defined considering the resources required, the schedule and budget; and the development and quality plans have been correctly determined. Software Functional Quality − It reflects how well it satisfies a given design, based on the functional requirements or specifications. in order to understand, assess, manage, engineer, learn, improve, etc. Factors such as increasing importance of quality assurance, customer centric production, and changes in standards and regulations are driving the demand for quality management software (QMS). $$\small Programmer\:Productivity = \frac{LOC \: produced }{Person \:months \:of \:effort}$$, $\small Module\:Defect\:Density = \frac{Number \:of\:defects}{Module \:size}$, $$\small Defect\:Detection\:Efficiency = \frac{Number \:of\:defects\:detected}{Total \:number \:of\:defects}$$, $\small Requirement\:Stability = \frac{Number \:of\:initial\:requirements}{Total \:number \:of\:requirements}$, $\small Test\:Effectiveness\:Ratio = \frac{Number \:of\:items\:covered}{Total \:number \:of \:items}$, $\small System\:spoilage = \frac{Effort \:spent\:for\:fixing\:faults}{Total \:project \:effort}$. Different possible components of a productivity model can be expressed in the following diagram. $$BMI = \frac{Number \: of \: problems \: closed \: during \:the \:month }{Number \: of \: problems \: arrived \: during \:the \:month} \times 100\%$$. The development life cycle stage components detect design and programming errors. Initiating and managing activities to improve and increase the efficiency of software maintenance and SQA activities. They include −. Thus, the process change can affect the organization and the project as well as the process. Process metrics − These characteristics can be used to improve the development and maintenance activities of the software. The higher the value of the metric, the more effective the development process and the fewer the defects passed to the next phase or to the field. When the target unit of assessment is at the project level, the assessment should include all meaningful factors that contribute to the success or failure of the project. Space complexity − The resource is computer memory. A simple example of the defined process is described in the following figure. The application of TQM to software: Software development is a process in which the developer precisely converts the requirement specifications into software products. The following direct measures are commonly used in software engineering. It can be any input for the software production. These factors are as follows −. [3] However, software's quality and fit-for-purpose status can still be realized in various ways depending on the organization and type of realized project. It ensures that developed software meets and complies with the defined or standardized quality specifications. This metric can be calculated for the entire development process, for the front-end before code integration and for each phase. [1][2] (Some like Sommerville link these responsibilities to quality assurance rather than call it quality control. Global flow − If information flows from one module to another through a global data structure. Typical goals are expressed in terms of productivity, quality, risk, customer satisfaction, etc. For example: If most of the data is present on the left-hand side of the mean, then we can say that the distribution is skewed to the left. [3]) These checks are optimally separate from the development team so as to lend more of an objective view of the product to be tested. It is a simple count of reported problems that remain at the end of each month or each week. The head of the SQA unit is responsible for all the quality assurance tasks performed by the SQA unit and its sub-units. Publication of the TickIT Guide, which supports the software industry’s efforts to spread ISO 9001 certification. These include adapting the current software to additional circumstances and customers without changing the software. Here, the user has to give a number by comparing the different options. Based on the following table, an ILF that contains 10 data elements and 5 fields would be ranked as high. If BMI is less than 100, then the backlog increased. Qualityze serves the purpose of quality management software for pharmaceutical industry quite well. These components can be classified into the following six classes −. There are three development products whose size measurement is useful for predicting the effort needed for prediction. This model classifies all software requirements into 11 software quality factors. Get Started for FREE Sign up with Facebook Sign up with Twitter I don't have a Facebook or a Twitter account. Because a large percentage of programming defects is related to design problems, conducting formal reviews, or functional verifications to enhance the defect removal capability of the process at the front-end reduces error in the software. According to the way in which data is moving through the system, it can be classified into the following −. At level 1, the project is likely to have ill-defined requirements. For example − This software performs well. The GQM approach provides a framework involving the following three steps −, Listing the major goals of the development or maintenance project, Deriving the questions from each goal that must be answered to determine if the goals are being met, Decide what must be measured in order to be able to answer the questions adequately. Validating prediction systems in a given environment is the process of establishing the accuracy of the prediction system by empirical means, i.e. The process will act as sensors and monitors, and we can change the process significantly in response to warning signs. It places the elements in an ordered classification scheme. Performance of audit-based assessments of software quality systems and consultation to organizations on the improvement of software development and maintenance processes in addition to their management. These activities focus on determining the defects in the actual products produced. For more than two variables, multivariate regression can be used. It also deals with the time between recharging of the system’s portable units, such as, information system units located in portable computers, or meteorological units placed outdoors. Most quality models include reliability as a component factor, however, the need to predict and measure reliability has led to a separate specialization in reliability modeling and prediction. It is a measurement mapping that preserves ordering, the size of intervals between the entities and the ratio between the entities. Normally, a software is developed for a contract negotiated with a customer or for an internal order to develop a firmware to be embedded within a hardware product. Some metrics belong to multiple categories. EI − The number of external inputs. It involves product-focused action. It measures the defects relative to the software size expressed as lines of code or function point, etc. In this case we have chosen American multinational technology and consulting firm which has its head office at Armonk, New York (IBM). Data can be defined according to the following points −, Collection of data requires human observation and reporting. When used within a quality management system, such an approach emphasizes the importance of the following −. The structure of SQA unit varies by type and size of the organization. An activity or set of activities using resources, and managed in order to enable the transformation of inputs into outputs, can be considered as a process. They include: The resulting numbers (Unadjusted FP) are grouped into Added, Changed, or Deleted functions sets, and combined with the Value Adjustment Factor (VAF) to obtain the final number of FP. Software quality assurance sets up an organized and logical set of organizational processes and deciding on that software development standards — based on industry best practices — that should be paired with those organizational processes, software developers stand a better chance of producing higher quality software. These are the measurements that can be measured without the involvement of any other entity or attribute. Examples include the number of software developers, the staffing pattern over the life cycle of the software, cost, schedule, and productivity. Another type of these requirements deals with automatic diagnostic checks applied by the maintenance technicians to detect the causes of software failures. It includes processing capabilities (given in MHz), its storage capacity (given in MB or GB) and the data communication capability (given in MBPS or GBPS). However, the three core components otherwise remain the same. from the viewpoint of the developer, manager, customer, etc. How frequently are transactions executed daily, weekly, monthly, etc.? Software quality assurance standards can be classified into two main classes −, Software quality assurance management standards, including certification and assessment methodologies (quality management standards), Software project development process standards (project process standards). Certification, which is granted after the periodic audits, will be valid only until the next audit, and therefore must be renewed. With regard to data collection, the CBA IPI relies on four methods −, The Standard CMMI Assessment Method for Process Improvement (SCAMPI) was developed to satisfy the CMMI model requirements (Software Engineering Institute, 2000). Assuring an acceptable level of confidence that the software will conform to functional technical requirements. To boost the decision of a particular investigation technique, the goal of the research should be expressed as a hypothesis we want to test. This metric is needed because development organizations cannot investigate and fix all the reported problems immediately. These are more closely associated with process and product metrics than with project metrics.


text:
# Variation of system energy in Canonical Ensemble

Tags:
1. Sep 22, 2015

### devd

A system is in contact with a reservoir at a specific temperature. The macrostate of the system is specified by the triple (N,V,T) viz., particle number, volume and temperature.

The canonical ensemble can be used to analyze the situation. In the canonical ensemble, the system can exchange energy with the reservoir, with the system energy varying, in principle, from zero to infinity.

I don't understand how the system energy can vary once we specify (N,V,T). The internal energy, after all, can be considered to be a function of T and V, U=U(T,V).

Surely, we're not talking about the fluctuation, i.e. the standard deviation in energy, since that is very small for systems with large N. Where am I going wrong?

2. Sep 22, 2015

### lautaaf

The $\underline{\text{expected value}}$ of the internal energy $<U>$ is the (macroscopical) quantity that depends of $N$,$T$.

3. Sep 22, 2015

### lautaaf

In order to find the density matrix, you have to minimize the Gibb's entropy: $$S= -k_B Tr(\hat \rho ln \hat \rho)$$
With the constraints: $$Tr(\hat \rho)=1$$ and $$<U>=Tr(\hat \rho \hat H) =E= constant$$
i.e, you let your system explore all the microstates (which include microstates with different energy) that are compatible with the macroscopic constraint $<U>=E=constant$

4. Sep 22, 2015

### devd

Yes, i think i get where i was going wrong. I was equating the thermodynamic notion of internal energy to the internal energy of one individual system, instead of the ensemble average. Thanks for the reply.

Just to be clear, there are multiple microstates with different U which correspond to the same macrostate defined by (N,V,T), right?
Also, this exchange of energy between the system and reservoir is basically due to statistical fluctuation, which i was ignoring?

5. Sep 22, 2015

### lautaaf

Exactly.
Moreover, you can prove that, if we define $\delta U = \frac{<U>-U}{U}$ then the standard deviation of $\delta U$ (the relative fluctuations) decay as:
$$<(\delta U)^{2}> \sim \frac{1}{N}$$


text:
Japan Journal of Industrial and Applied Mathematics

Preconditioned GMRES Methods for Least Squares Problems

Abstract

For least squares problems of minimizing $\|\boldsymbol{b}-A\boldsymbol{x}\|_2$ where $A$ is a large sparse $m\times n$ ($m\ge n$) matrix, the common method is to apply the conjugate gradient method to the normal equation $A^{\mathrm{T}} A\boldsymbol{x}=A^{\mathrm{T}} \boldsymbol{b}$. However, the condition number of $A^{\mathrm{T}} A$ is square of that of $A$, and convergence becomes problematic for severely ill-conditioned problems even with preconditioning. In this paper, we propose two methods for applying the GMRES method to the least squares problem by using an $n\times m$ matrix $B$. We give the necessary and sufficient condition that $B$ should satisfy in order that the proposed methods give a least squares solution. Then, for implementations for $B$, we propose an incomplete QR decomposition IMGS($l$). Numerical experiments showed that the simplest case $l=0$ gives the best results, and converges faster than previous methods for severely ill-conditioned problems. The preconditioner IMGS(0) is equivalent to the case $B=(\diag (A^{\mathrm{T}} A))^{-1} A^{\mathrm{T}}$, so $(\diag (A^{\mathrm{T}} A))^{-1} A^{\mathrm{T}}$ was the best preconditioner among IMGS($l$) and Jennings' IMGS($\tau$). On the other hand, CG-IMGS(0) was the fastest for well-conditioned problems.

Article information

Source
Japan J. Indust. Appl. Math., Volume 25, Number 2 (2008), 185-207.

Dates
First available in Project Euclid: 3 July 2008


text:
# Image warping using pixel normals

I want to insert a logo into an RGB image with proper 3D transformations. I can get the estimated pixel normals of this image using the code here: https://github.com/yindaz/surface_normal. Pixel normals are encoded into RGB channels as described as follows:

The estimated normal will be saved in 16-bit PNG format, where 0-65535 in R,G,B channel correspond to [-1, 1] for the X,Y,Z component of the normal vector. We use the camera coordinates defined as - X points to the camera right, Y points to the camera forward, and Z points to the camera upward. For example, right facing wall are very red, floor are very blue, and you rarely see green as it's parallel to the camera viewing direction.

How can I transform the 2D logo to put it onto a planar surface in the image? I have tried to calculate the rotation angles of the normal to rotate it so that it has the same direction as the camera normal, and rotate the logo according to these angles, but I have no success so far.

As far as I understand from the quote above, normal of the untransformed logo should be $(x:0, y:-1, z:0)$ and let the pixel normal be $(x:a, y:b, z:c)$ with magnitude $M$. I rotate the logo around $X$ by $arccos(a/M)*180/pi$, around $Y$ by $arccos(b/M)*180/pi$, around $Z$ by $arccos(c/M)*180/pi$, and if $b < 0$ rotate around $Y$ by an additional 180 degrees.

Is there anything reasonable in these? Is it possible that I can do this using the estimated normals?

• You need to search for "Homography". Estimate a homography that places the image in the scene based on the normals and transform your image by the homography. – Wyck Jan 31 '18 at 15:35
• So it is possible to estimate a homography using normals, right? I have found homography estimation using two images or through finding vanishing points in a single image so far, but not using normals. Is there a specific term for this that I should search for? – groove Feb 1 '18 at 5:07
• I assume the normals help you estimate depth so that you get 3d points. You don't need that if the surface is planar though. Just identify the four corners of the rectangle that you want to map in the target image and use the homography. – Wyck Feb 2 '18 at 2:41
• This would work only if there is a rectangle and it can be found, right? I need to find areas with no semantically significant content, such as empty walls, and insert a logo there. I couldn't succeed using corner detection-based methods or homography estimation using parallel lines. Therefore I though surface normals can be useful since there is a normal for each pixel. – groove Feb 5 '18 at 13:03
• Normals are as useful as derivatives in Calculus. Much like how you can integrate derivatives to get the original function (+/- a constant), you can do something similar to your normals to get the original surface (+/- a global translation). But knowing the geometry of the surface doesn't describe WHERE you want to put your image on that surface. I feel like I still don't understand how you want to describe where the image goes. But I also feel like you can bypass the problem if you use the homography. – Wyck Feb 5 '18 at 14:18


text:
# Wydawnictwa / Czasopisma IMPAN / Annales Polonici Mathematici / Wszystkie zeszyty

## Almost homoclinic solutions for a certain class of mixed type functional differential equations

### Tom 100 / 2011

Annales Polonici Mathematici 100 (2011), 13-24 MSC: Primary 34C37; Secondary 34K13, 58E50. DOI: 10.4064/ap100-1-2

#### Streszczenie

We shall be concerned with the existence of almost homoclinic solutions for a class of second order functional differential equations of mixed type: $\ddot{q}(t)+V_{q}(t,q(t))+u(t,q(t),q(t-T),q(t+T))=f(t)$, where $t\in\mathbb R$, $q\in\mathbb R^{n}$ and $T>0$ is a fixed positive number. By an almost homoclinic solution (to $0$) we mean one that joins $0$ to itself and $q\equiv 0$ may not be a stationary point. We assume that $V$ and $u$ are $T$-periodic with respect to the time variable, $V$ is $C^{1}$-smooth and $u$ is continuous. Moreover, $f$ is non-zero, bounded, continuous and square-integrable. The main result provides a certain approximative scheme of finding an almost homoclinic solution.

#### Autorzy

• Joanna JanczewskaFaculty of Technical Physics and Applied Mathematics
Gdańsk University of Technology
Narutowicza 11/12
80-233 Gdańsk, Poland
and
Institute of Mathematics


text:
# Revision history [back]

### Select elioment of a list through the value of an operator

Here follow an occurence of a Fourier-Motzkin elimination.

fmf=[z<=4*x_1+ 3*x_2, x_1 + 3*x_2 <= 2100, 4*x_1+ 2*x_2<=1900, x_1<=200,x_2>=300, x_1 >=0,x_2 >=0]
fmf
fm_sol=solve_ineq(fmf,[x_1, x_2,z])
fm_sol

How can I select only the elements of the list where the operator for z is == ?

### Select elioment of a list through the value of an operator

Here follow an occurence of a Fourier-Motzkin elimination.

fmf=[z<=4*x_1+ 3*x_2, x_1 + 3*x_2 <= 2100, 4*x_1+ 2*x_2<=1900, x_1<=200,x_2>=300, x_1 >=0,x_2 >=0]
fmf
fm_sol=solve_ineq(fmf,[x_1, x_2,z])
fm_sol

How can I select only the elements of the list where the operator for z is == ?

### Select elioment element of a list through the value of an operator

Here follow an occurence of a Fourier-Motzkin elimination.

fmf=[z<=4*x_1+ 3*x_2, x_1 + 3*x_2 <= 2100, 4*x_1+ 2*x_2<=1900, x_1<=200,x_2>=300, x_1 >=0,x_2 >=0]
fmf
fm_sol=solve_ineq(fmf,[x_1, x_2,z])
fm_sol

How can I select only the elements of the list generated by fm_sol where the operator for z is == ?


text:
# Moiré patterns in a pair of hexagonal lattices

A Moiré pattern is an interference pattern that occurs when two grids of repeating lines or shapes are rotated by a small amount relative to one another (oblig. xkcd).

A recent application concerned the experimental verification of Hofstadter's butterfly: Douglas Hofstadter (he of Gödel, Escher, Bach fame) predicted in a 1976 paper that the energy structure of the electrons in a two-dimensional lattice of atoms in a magnetic field has a fractal structure. The problem with the experimental verification of this prediction is that the lattice dimensions of a typical solid material are so small that it would be unfeasible to generate the necessary magnetic field to make the energy structure measurable.

In 2013, three research groups were able to verify the claim by placing a layer of graphene on a substrate layer of boron nitride (which has a very similar hexagonal atomic structure), at such an angle that the resulting Moiré pattern had a much larger repeating length scale.

The code below generates an SVG image of two hexagonal lattices, rotated by a specified amount, which shows the repeating Moiré pattern that results.

import numpy as np

# Image dimensions and grid lattice size, a.
width, height = 600, 400
a = 5

# a.sin(60) and a.cos(60)
gx, gy = a * np.sqrt(3)/2, a/2
# The number of unit cells, horizontally and vertically.
nx, ny = int(width / 2 / gx)+1, int(height / (a + gy))+1
# We'll need the coordinates of the centre of the image when we come to rotate.
cx, cy = width / 2, height / 2

def svg_line(x0, y0, x1, y1, th=0, cls=None):
"""Return the SVG for a single, line possibly rotated by th radians."""

def rotate(x, y, th):
"""Rotate the coordinates (x,y) about the centre (cx,cy)."""
c, s = np.cos(th), np.sin(th)
xp, yp = x-cx, y-cy
x, y = c*xp - s*yp, s*xp + c*yp
return x+cx, y+cy

if th != 0:
x0, y0 = rotate(x0, y0, th)
x1, y1 = rotate(x1, y1, th)

# If an SVG class has been provided, add it to the line element.
s_cls = 'class="{}"'.format(cls) if cls else ''
return '<line x1="{}" y1="{}" x2="{}" y2="{}" {}/>'.format(
x0, y0, x1, y1, s_cls)

def add_unit_cell(s, x0, y0, th=0, cls=None):
"""Add a unit cell from the lattice to the SVG output.

The "unit cell" consists of the arrangement of lines: \ /
|
centred at the vertex where they meet. th is the angle of rotation, in
radians and cls is an optional SVG class to add to the <line> element.

"""

s.append(svg_line(x0, y0, x0, y0+a, th, cls))
s.append(svg_line(x0, y0, x0-gx, y0-gy, th, cls))
s.append(svg_line(x0, y0, x0+gx, y0-gy, th, cls))

def svg_preamble(s):
"""The usual SVG preamble and style definitions."""

s.append('<?xml version="1.0" encoding="utf-8"?>')
s.append('<svg xmlns="http://www.w3.org/2000/svg"\n' + ' '*5 +
.format(width, height))
s.append("""<defs>
<style type="text/css"><![CDATA[
line {
stroke-width: 2px;
stroke: #000;
}
.lattice1 {
stroke: #824e4e;
}
.lattice2 {
stroke: #4a5f70;
}
>]]></style>
</defs>
""")

s = []
svg_preamble(s)

# Angle of rotation of the lattices (degrees)
th = 10
# An internal variable to offset every other row of the unit cells when drawing
_ph = 0
for iy in range(ny):
_ph = 0 if _ph else gx
for ix in range(nx):
add_unit_cell(s, ix*2*gx + _ph, iy*(a+gy), cls='lattice1')

s.append('</svg>')

with open('lattice.svg', 'w') as fo:
print('\n'.join(s), file=fo)

Current rating: 5

#### Alfonsi Jessica 3 months, 2 weeks ago

Wonderful blog! It makes Python even more appealing to me, I just need a good problem to start learning by coding.
Have a look at my Mathematica Demonstration I published 5 years ago
http://demonstrations.wolfram.com/MoirePatternsAndCommensurabilityInRotatedGrapheneBilayers/

Currently unrated

#### christian 3 months, 2 weeks ago

Thank you! I think we may have been reading the same papers about rotated graphene layers. I was going to write something about Hofstadter's Butterfly as well but never got round to it...

Currently unrated

#### Alfonsi Jessica 3 months, 1 week ago

I have been thinking often about plotting Hofstadter's butterfly for different lattices, square and hexagonal, but now I'm looking into other systems. There's a recent IOP ebook https://iopscience.iop.org/book/978-1-6817-4117-8