text:
I can program and like games

### Finding Floating Point Numbers for Exact Math

For the second time in my career, I ran into a problem where it’s actually useful to know how floating point numbers work. (first time was here) The problem is that sometimes you want floating point numbers that add well. So that you have the associativity guarantee that (a + b) + c == a + (b + c).

The use case here was for A* path finding, but this may also be applicable in other situations. I’ll explain a bit more of A* later, but as a motivation you just have to know that you can speed up A* a lot if you can reason about all the nodes in your graph that have the same “cost.” But since the “cost” is a sum of a lot of different floats, they will rarely have exactly the same value. At that point you could fudge numbers and say that “if two numbers are equal to within 0.0001, treat them as equal” but when you do that it’s easy to accidentally define an ordering that’s not a strict weak ordering. And I have literally seen a crash caused by an incorrect float ordering that was very rare and very hard to track down. So once bitten twice shy, I wanted to see if I couldn’t just force floating math to be exact. And turns out you often can.

### Where do top scientists come from? And what do taxes have to do with it?

I was reading this article recently, which talks about “Where star scientists choose to locate: the impact of US state taxes” It’s a summary of a paper about “the effect of state taxes on the geographical location of top earners.”

It’s a very interesting idea: The problem is that states often lower taxes with the hope of attracting business or talent, but there is very little evidence about whether that actually works. So the authors of that paper decided to find a group of influential people who are somewhat easy to track: people who apply for lots of patents, the so called “star scientists” from the title. So the authors built a huge database, tracking where the top 5% of scientists who applied for the most patents had moved to over the years.

And the authors claim that they found pretty clear evidence that people like to move from high-tax states to low-tax states, so the conclusion is that if you want to attract top scientists, you should lower taxes.

Except, I dug through the data and I found the opposite. Yes, top scientists do move to states that have lower taxes, but high tax states have such a large lead in the number of scientists, that that little bit of migration doesn’t matter. But we’ll have to get to that conclusion one step at a time.

### Games Are About Personal Development

Here’s an angle on the fundamental reason for why we play games: They are about personal development, learning about ourselves and about the world. This may not be a new angle, but I haven’t heard it stated this explicitly. Instead I have heard people say stupid things like “games teach hand-eye-coordination” which is true, but also bullshit because why would you spend this much time training your hand-eye-coordination? No I claim that games teach import life lessons, and that that is the fundamental reason why we play games.

I’m going to talk about video games, but this is also about games in general. Why do kids play with dolls? Because they want to learn about family life. (or about conflicts when playing with action figures) This is not explicit learning like we learn from a teacher, but you act out situations and adjust your behavior depending on how your play partner reacts. Why do we send our kids to football practice? Not because we think that they need to learn the valuable skill of kicking a ball into a net. No it’s because we want them to learn about working in teams and about pacing themselves and about playing fair and all that.

The things we learn are obvious in those scenarios. It’s well known that it’s important for kids to play in order to figure out how to act in the world in a safe environment. But I claim that the same thing is true for video games, and my example will be Super Mario World.

### Evidence For How To Make Great Games

Earlier this year I gave a talk about the Game Outcomes Project. I called that talk “Evidence For How To Make Great Games” because I think the Game Outcomes Project is the best data we have for what teams do that make great games. I wasn’t involved in the Game Outcomes Project, I just gave a talk about it because I really like it. Also I wanted to focus on different things than what they focused on in their own write-ups and talks.

People who saw the talk said that they really liked it, and they keep on telling me how much they liked it. So I decided to record the talk again and upload it.

The pitch for the talk is that the results of the Game Outcomes Project is the best evidence we have for what makes great game development teams and what makes bad game development teams. And I think that every game developer should know this stuff. So I talk about what you should focus on when making a game, and I give advice for how to get there. So the game outcomes project found “really successful teams do X” and I present that, and then also have a section at the end of the talk where I say “here is how you can actually get good at doing X.” Here is the talk:

### Collected Advice for Doing Scientific Research

I’ve been collecting advice from various sources about how to do research. I can’t find a good place that collects this, so let’s see if I can create one.

The idea is that this should be something similar as George Polya’s “How to Solve It” but for doing research instead of solving problems. There is a lot of overlap between those two ideas, so I will quote a lot from Polya, but I will also add ideas from other sources. I should say though that my sources are mostly from Computer Science, Math and Physics. So this list will be biased towards those fields.

My other background here is that I work in video game AI so I’ve read a lot of AI literature and have found parallels between solving AI problems and solving research problems. So I will try to generalize patterns that AI research has found about how to solve hard problems.

A lot of practical advice will be for getting you unstuck. But there will also be advice for the general approach to doing research.

### Less Weird Quaternions

I’ve always been frustrated by how mysterious quaternions are. They arise from weird equations that you just have to memorize, and are difficult to debug because as soon as you deviate too far from the identity quaternion, the numbers are really hard to interpret. Most people implement quaternions once and then treat them as a black box forever after. So I had put quaternions off as one of those weird complicated 4D mathematical constructs that mathematicians sometimes invent that magically works as long as I don’t mess with it.

That is until recently, when I came across the paper Imaginary Numbers are not Real – the Geometric Algebra of Spacetime which arrives at quaternions using only 3D math, using no imaginary numbers, and in a form that generalizes to 2D, 3D, 4D or any other number of dimensions. (and quaternions just happen to be a special case of 3D rotations)

In the last couple weeks I finally took the time to work through the math enough that I am convinced that this is a much better way to think of quaternions. So in this blog post I will explain…

• … how quaternions are 3D constructs. The 4D interpretation just adds confusion
• … how you don’t need imaginary numbers to arrive at quaternions. The term $\sqrt{-1}$ will not come up (other than to point out the places where other people need it, and why we don’t need it)
• … where the double cover of quaternions comes from, as well as how you can remove it if you want to (which makes quaternions a whole lot less weird)
• … why you actually want to keep the double cover, because the double cover is what makes quaternion interpolation great

Unfortunately I will have to teach you a whole new algebra to get there: Geometric Algebra. I only know the basics though, so I’ll stick to those and keep it simple. You will see that the geometric algebra interpretation of quaternions is much simpler than the 4D interpretation, so I can promise you that it’s worth spending a little bit of time to learn the basics of Geometric Algebra to get to the good stuff.

### I Wrote The Fastest Hashtable

I had to get there eventually. I had a blog post called “I Wrote a Fast Hashtable” and another blog post called “I Wrote a Faster Hashtable.” Now I finally wrote the fastest hashtable. And by that I mean that I have the fastest lookups of any hashtable I could find, while my inserts and erases are also really fast. (but not the fastest)

The trick is to use Robin Hood hashing with an upper limit on the number of probes. If an element has to be more than X positions away from its ideal position, you grow the table and hope that with a bigger table every element can be close to where it wants to be. Turns out that this works really well. X can be relatively small which allows some nice optimizations for the inner loop of a hashtable lookup.

If you just want to try it, here is a download link. Or scroll down to the bottom of the blog post to the section “Source Code and Usage.” If you want more details read on.

### Faster Sorting Algorithm Part 2

This is a follow up to my previous blog post about writing a faster sorting algorithm. I’m using this as a chance to go into detail on topics that I was asked about in the comments: I’ll clear up some misunderstandings and go into future work that needs to happen with this algorithm.

Somebody was nice enough to link my blog post on Hacker News and Reddit. While I didn’t do that, I still read most of the comments on those website. For some reasons the comments I got on my website were much better than the comments on either of those websites. But there seem to be some common misunderstandings underlying the bad comments, so I’ll try to clear them up.

### random_seed_seq: A small utility to properly seed random number generators in C++

Some libraries are so small they’re almost not worth sharing. But the C++ standard has a giant hole in that it doesn’t provide an easy way to quickly generate truly random numbers: It has std::mt19937_64 which quickly generates pseudo-random numbers, and it has std::random_device, which slowly generates truly random numbers.

The easiest way to quickly generate truly random numbers is to use a std::random_device to seed a std::mt19937_64. That way we pay a one-time cost of using random device to generate a seed, and then have quick random numbers after that. Except that the standard doesn’t provide a way to do that. In fact it’s more dangerous than that: It provides an easy wrong way to do it (use a std::random_device to generate a single int and use that single int as the seed) and it provides a slow, slightly wrong way to do it. (use a std::random_device to fill a std::seed_seq and use that as the seed) There’s a proposal to fix this, (that link also contains reasons for why the existing methods are wrong) but I’ve actually been using a tiny class for this:

### I Wrote a Faster Sorting Algorithm

These days it’s a pretty bold claim if you say that you invented a sorting algorithm that’s 30% faster than state of the art. Unfortunately I have to make a far bolder claim: I wrote a sorting algorithm that’s twice as fast as std::sort for many inputs. And except when I specifically construct cases that hit my worst case, it is never slower than std::sort. (and even when I hit those worst cases, I detect them and automatically fall back to std::sort)

Why is that an unfortunate claim? Because I’ll probably have a hard time convincing you that I did speed up sorting by a factor of two. But this should turn out to be quite a lengthy blog post, and all the code is open source for you to try out on whatever your domain is. So I might either convince you with lots of arguments and measurements, or you can just try the algorithm yourself.

Following up from my last blog post, this is of course a version of radix sort. Meaning its complexity is lower than O(n log n). I made two contributions:

1. I optimized the inner loop of in-place radix sort. I started off with the Wikipedia implementation of American Flag Sort and made some non-obvious improvements. This makes radix sort much faster than std::sort, even for a relatively small collections. (starting at 128 elements)
2. I generalized in-place radix sort to work on arbitrary sized ints, floats, tuples, structs, vectors, arrays, strings etc. I can sort anything that is reachable with random access operators like operator[] or std::get. If you have custom structs, you just have to provide a function that can extract the key that you want to sort on. This is a trivial function which is less complicated than the comparison operator that you would have to write for std::sort.

If you just want to try the algorithm, jump ahead to the section “Source Code and Usage.”


text:
# vim ruby mismatch on Mac High Sierra

I recently upgraded my Mac to High Sierra

After the upgrade, Macvim (mvim) stopped working because (I assume) HighSierra upgraded ruby to 2.3, and mvim is using ruby 2.0.

Following is the error I am getting

Referenced from: /usr/local/Cellar/macvim/8.0-134/MacVim.app/Contents/bin/../MacOS/Vim

Do I need to compile vim again or is there any way to fix this problem?

• You could try changing the dyld to the new one in the Vim binary with install_name_tool... – l'L'l Nov 14 '17 at 5:57
• Thank you very much. It solved my problem. – user2960993 Nov 14 '17 at 6:07

This worked for me. YMMV  brew uninstall --purge macvim brew install macvim

[amended] Regarding the upgrade alone option, that may work if you don't have the current version already. The --purge forces a removal and complete reinstall. Also, the build from source option, while it may work, you now own your own mac-vim static snapshot and need to maintain it manually as the app evolves. I'd avoid building from source unless you have very specific needs.

• Did the trick for me as well. Updating from GUI as suggested in other post did not work for me. Thanks, Bruce! – bduhbya Nov 16 '17 at 21:02
• Yep, this one helped me as well! But, I should add that I also had to run GUI Xcode app itself at least once before installing vim, since it needed to install some additional components. See: github.com/macvim-dev/macvim/issues/546 – smileart Nov 22 '17 at 16:36
• This works for vim too, if one insn't using macvim. – Chris Townsend Feb 1 '18 at 22:03
• In the entire history of the git source for brew I can't find mention of a --purge option. Did you mean --force? – Skrat Feb 12 at 17:48

First find where is the right libruby path, mine is: /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/libruby.2.3.0.dylib

It is common that the version has changed.

Find out where the absolute path of vim is:

$which vim vim: aliased to your-path-of-vim  change dyld: sudo install_name_tool -change /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/libruby.2.0.0.dylib /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/libruby.2.3.0.dylib your-path-of-vim  • This worked with a minor difference. Instead of using which vim, use the path indicated in error. In my case, it was /usr/local/Cellar/macvim/8.1-151/MacVim.app/Contents/bin/../MacOS/Vim – Anshul Nov 5 '18 at 8:39 • Worked for me -- thanks! – Bryce Johnson Feb 27 at 18:10 ➜ ~ vim dyld: Library not loaded: /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/libruby.2.0.0.dylib Referenced from: /usr/local/bin/vim Reason: image not found  I tried brew uninstall install update... All failed. And install_name_tool works for me. install_name_tool -change /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/libruby.2.0.0.dylib /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/libruby.2.3.0.dylib /usr/local/bin/vim  • I was in the same situation with none of the other solutions working, but the install_name_tool worked for me, thank you! – J Ellis Apr 22 '18 at 22:26 • After tried, I got this error: error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: input file: /usr/local/Cellar/macvim/8.0-143/MacVim.app/Contents/bin/mvim is not a Mach-O file – teleme.io Jul 23 '18 at 4:42 • This worked for me, but it was a slightly different binary location. install_name_tool -change /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/libruby.2.0.0.dylib /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/libruby.2.3.0.dylib /usr/local/Cellar/macvim/8.1-151/MacVim.app/Contents/MacOS/MacVim and install_name_tool -change /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/libruby.2.0.0.dylib /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/libruby.2.3.0.dylib /usr/local/Cellar/macvim/8.1-151/MacVim.app/Contents/MacOS/Vim – Juan Chanco Nov 2 '18 at 13:35 • Important to note that /usr/local/bin/vim should be replaced with whatever the error says is the referencing file. In my case, it was /usr/local/Cellar/macvim/8.1-151/MacVim.app/Contents/bin/../MacOS/Vim – abeger Nov 2 '18 at 15:09 This worked for me: $ brew update

$brew upgrade macvim  actual command output: [bin]$ brew upgrade macvim
==> Upgrading 1 outdated package, with result:
macvim 8.0-142
######################################################################## 100,0%
==> ./configure --with-features=huge --enable-multibyte --with-macarchs=x86_64 --enable-perlinterp --enable-rubyinterp --enable-tclinterp --enable-ter
==> make
🍺  /usr/local/Cellar/macvim/8.0-142: 2,147 files, 34.2MB, built in 3 minutes 15 seconds
[bin]\$

What helped in my case was:

brew link --overwrite vim

In your case, it might be useful to use the same command for MacVim

brew link --overwrite macvim

Tried to use brew upgrade macvim an error was raised and the failed. What worked for me is to open the MacVim in a normal window (not a iTerm or terminal) the graphic version noticed that a newer version is available and asked if I want to update. Clicked yes, please and done. macvim in terminal or iTerm works just fine (as well as the "window" version).

format: plain


text:
# Why does the TeX scanner process tokens for register numbers and macro names differently?

This is a "How does TeX know when to stop" question.

# Case 1

Note: I have since found out that my understanding here is wrong. See SPOILER and answers.

When TeX is constructing a macro name, it consumes all category 11 tokens until it scans a non-category 11 token. The non-category 11 token is

• Scanned: TRUE
• Typeset: FALSE

# Case 2

When TeX is constructing a register value, it consumes all tokens of category 12 (number) until it finds a non-category 12 token. The non-category 12 token is somehow both

• Scanned: TRUE
• Typeset: TRUE

# Code

\documentclass{article}
\usepackage{fontspec}% xelatex
\begin{document}
\section{Macro Name Scanning}
\def\macro{123}
\count1=\macro
4
This is a register: \the\count1. Notice that the first 4 was consumed by the TeX scanner as expected.

\section{Register Value Scanning}
\count1=42
\advance\count1 by1000The < The \textit{T} in \textit{The} should have been consumed by the TeX scanner before being typeset. Where does \textbackslash relax come into play here?% Why is the T not consumed by the number scanner (catcode 12)
\end{document}

# Notes

I expected that I would need a \relax before the T in case 2 as I do after \macro in case 1.

# SPOILER ALERT

My confusion is explained in the answers below, but I'd like to point out something I learned:

A difference between macro construction (scanning csnames) and register value construction (scanning number values) is:

• control sequence name/macro name scanning: TeX scans characters bytes (tokens not yet created, rather being actively created at this step)
• register value scanning: TeX scans character tokens (tokenization has already happened. A character token is an integer pair consisting of (character code, cat code))

This has important implications.

• control sequence name/macro name scanning sees all character bytes of a file and applies certain TeX rules of tokenization (e.g. when to leave out certain bytes like spaces or newlines after macros)
• register value scanning never sees character bytes, rather the resulting character token stream after bytes have been processed. A macro token is expanded when encountered and newline character bytes have already been converted to catcode 10 (i am not sure where catcode 5 comes into play if at all here). There is a space rule at this stage too. Catcode 10 tokens are removed when read, and not put back into the input stream.

P.S. I sure hope I have the details right!

• The last paragraph is quite misleading. – egreg Jan 19 '17 at 15:34
• @egreg How so? I tested The\macro story goes... and The\count1=123 story goes.... Both spaces (after \macro and \count1=123) were missing in the resulting PDF. Maybe it has something to do with the new line? – Jonathan Komar Jan 19 '17 at 15:39
• Try \count1=\macro 4 and look at the value of \count1. Then try \count1=\macro\space 4. – egreg Jan 19 '17 at 15:41
• There is no space: try x\count1=\macro\space 4 and you'll see no space between x and 4 in the printout. Note that \space is not “backslash space”. – egreg Jan 19 '17 at 16:00
• x\count1=\macro\space 4 is seven tokens. The control sequence \space is tokenized. – egreg Jan 19 '17 at 16:09

## 3 Answers

I don't think that your description of scanning for a macro name is accurate (although I do not know what you mean by typeset:false, so that may or may not be true)

Note that scanning for macro names happens before tokenisation (or rather during it) as it is the determinisation of whether to make a csname token. After the character (not token) that has catcode 0 is seen then if the next character has catcode 11 all further catcode 11 characters are scanned as part of the csname, the first non-catcode 11 character is consumed and not tokenised if it would make a token of catcode 10 (space) otherwise it is put back into the input stream.

Numbers are part of tex's inner execution (what the TeXBook calls the stomach) so happen after tokenisation and after macro expansion. If TeX is parsing for a <number> then it will expand all tokens until it finds an unexpandable token that is part of a number syntax (so catcode 11 or 12 digits, or after " also hex letters A-F and other forms such as alphabetic character code numbers such as a or \a The first non expandable token that is not part of the syntax for <number> is always returned (as a token) to the input stream unless it is a space token, which is your T here.

• @macmadness86 only very indirectly, obviously anything that is typeset must have been passed to tex's typesetting routines so must have been tokenised at some point, but other than that typesetting is in no way involved here. consider def\foo{\abc .} and \def\foo{\abc.} nothing is typeset (unless \foo gets used later in a typesetting context) but they are equivalent definitions as the space character is never tokenised due to the rules above so in either case the replacement text of \foo is the two tokens \abc and . – David Carlisle Jan 19 '17 at 10:28
• Thanks for answering. I was about to rephrase my first comment, because as I think about it, I understand more :) I should have left it. Bad timing. – Jonathan Komar Jan 19 '17 at 10:29
• +check because the last sentence was so enlightening: "The first non expandable token that is not part of the syntax for <number> is always returned (as a token) to the input stream unless it is a space token, which is your T here." Of course egreg's explanation of "radix" indicators is essential too. – Jonathan Komar Jan 19 '17 at 11:20
• @macmadness86 the important part is that you gave the tick to me not egreg, he's always so pleased for me when that happens:-) – David Carlisle Jan 19 '17 at 11:53
• You're probably right. It was the right thing to do--you, with a mere 366,000 points. In summary: An important difference between how TeX scans a csname vs how TeX scans a register value is that for csnames, TeX consumes a space and stops scanning. On the other hand, when TeX scans numbers for a register value, it scans spaces but keeps scanning for more numbers. The only letters that can be numbers are A,B,C,D,E,F when using them to represent hexidecimal numbers. I hope this is a correct assessment of the process. – Jonathan Komar Jan 19 '17 at 12:01

Your assumptions are wrong. When doing an assignment to a count register, TeX obeys the syntax

<count register><optional equal sign><number>

where <count register> can be an internal numeric register (for instance \hyphenpenalty), a \countdef token (after \newcount\foo or a direct \countdef\foo=1) or \count<number>.

The <optional equal sign> can be empty, a space token, or =12 surrounded by (optional) space tokens (well it's a bit more complicated, but not really relevant).

A <number> consists of an optional radix indicator

 ' "

and then a sequence of digits in the corresponding radix. For the case of a backquote, the “digit” is a single character or a control sequence of length one (alphabetic constant). For ' the digits are among 01234567 (with category code 12); for " the digits are 0123456789ABCDEF (where the letters can have category code 11 or 12, but 0123456789 must have category code 12). Without an indicator, decimal is assumed and the admissible digits are 0123456789 with category code 12.

The <number> ends when, after macro expansion, TeX finds a token that cannot be interpreted as a suitable digit for the chosen radix. If this token is a space, it is gobbled.

Your example

\def\macro{123}

\count1=\macro
4

assigns to \count1 the integer 1234, because there is nothing between \macro and 4, as spaces are ignored during tokenization after a control word (and the end-of-line character gets converted to a space during the same tokenization phase). If you want to assign 123 and print 4, you have to say

\count1=\macro\space 4

(whether 4 is on the next line is irrelevant).

The syntax for \advance is similar (instead of = there is the optional by); so in

\advance\count1 by 1000T

the search for digits will stop at T, but it would stop at ! as well.

Some more words on the assignments above. With

\count1=\macro
4

TeX sees six tokens

\count • 1 • = • \macro • 4 • ⍽

(spaces are just for clarity, • separates tokens from each other). Exactly the same would be found with

\count1=\macro4
\count1=\macro 4

because of the rules about tokenization. There is no space token after \macro. The final ⍽ denotes the space token deriving from the end-of-line, which is tokenized because it doesn't get ignored during macro name constructions.

Since \count is unexpandable, TeX “executes” it, which means it has to perform an assignment, so a <number> is searched for; the following token is 1 which is right; the next token is =, so the search for digits stops and TeX knows it has to assign a value to \count1. The = is gobbled because it optional and TeX starts searching for a number. The first token is expandable, so TeX expands it getting

1 • 2 • 3 • 4 • ⍽

Now ⍽ stops the search for digits and the assignment can be performed. This space token is gobbled by rule.

Let's examine

\count1=42
\advance\count1 by1000The

and divide it into tokens:

\count•1•=•4•2•⍽•\advance•\count•1•⍽•b•y•1•0•0•0•T•h•e

(no typographic space for compactness, • still separates tokens from each other). The assignment of 42 to \count1 is performed as before and the space token is gobbled. We remain with

\advance•\count•1•⍽•b•y•1•0•0•0•T•h•e

The execution of \advance makes TeX to search for an appropriate register name, which it finds as \count1 (the space token is ignored after the lookup for the register's number). Next by is gobbled because it is an optional keyword and we remain with

1•0•0•0•T•h•e

TeX is looking for a <number> so it does as before, with macro expansion. However there is no macro to expand and the search for digits stops at the first token that is not a digit for the radix 10, so at T. The assignment is performed and \count1 is assigned the value 1042. Next the token T is reexamined and it will start a paragraph.

Be careful because something like

\count1="1000At this point

will assign to \count1 the value 65546 and start a paragraph with t.

Always end constants with an unexpandable token that's not a digit. A space is good for this as it will be ignored.

• The hard part for me is reconciling "there is nothing between \macro and 4" (hence 4 is not typeset, rather assigned to the register--because TeX will gobble the 4 while constructing the csname) with "the search for digits will stop at T" (hence T is typeset and NOT assigned to the register). I think David hit the nail on the head when he wrote "consumed and not tokenised [...] if it would make a token of catcode 10 (space) otherwise it is put back into the input stream". Perhaps my knowledge gap is at the tokenization stage. – Jonathan Komar Jan 19 '17 at 10:21
• @macmadness86 Why should TeX gobble the 4? If you type \macro4 you get 1234 – egreg Jan 19 '17 at 10:30

What do you mean by "constructing a macro name"?

At the time of reading/tokenizing input it is not important whether the control sequence token in question will denote a macro or a primitive or an undefined control sequence.

At the time of reading and tokenizing input, there are no tokens yet, thus names of control sequence tokens do not consist of whatsoever tokens. They are formed by characters (not character tokens). \string can be used for obtaining a sequence of corresponding character tokens. \csname..\endcsname offers a means of denoting control sequence tokens by means of sequences of expandable tokens and character-tokens.

Always take care about when it is appropriate to use the term "token" and when it is not appropriate to use the term "token" as the -eh- "object of contemplation" is not a token/is not something that came into being as a result of tokenization or expansion.

Control sequence tokens come in two flavours:
1. Control symbol tokens.
2. Control word tokens.

A sequence of two characters in the input where the first character's category code is 0 (escape) and the second character's category code is not 11 (letter) is taken for the identification of a control symbol token and therefore the corresponding control symbol token will be "placed" into the "stream of tokens".

A sequence of characters in the input where the first character's category code is 0 (escape) and all trailing characters of that sequence have category code 11 (letter) is taken for the identification of a control word token and therefore the corresponding control word token will be "placed" into the "stream of tokens".

IIRC the "nameless" control sequence token, denoteable via \csname\endcsname or via a character whose category code is 0 (escape) at the end of a line of input in situations where no endline-character gets attached, is also a control word token.

Control sequence tokens that come into being as a result of expanding \csname..\endcsname or \ifcsname..\endcsname and whose names form character-sequences other than just a single non-catcode-11-character are also control word tokens.

After reading/tokenizing a control word token, TeX' reading apparatus will switch to state S (skipping blanks). When unexpanded-writing a control word token, TeX will write/attach an additional space character.

After reading/tokenizing a control symbol token, TeX' reading apparatus will switch to state M (middle of line). When unexpanded-writing a ontrol symbol token, TeX will not write/attach an additional space character.

(Be aware that treatment at writing-time of a one letter control sequence token either as a control word token or as a control symbol token depends on the category code of the letter/character in question at writing-time.)

The last thing that will be gathered with a \count-assignment is a <number>.

The components of quantities subsumed under the term <number> are explained in Donald E. Knuth's The TeXbook, Chapter 24: Summary of Vertical Mode.

When gathering/reading the components of a <number>, expansion takes place unless TeX is in a situation where TeX does not expand tokens. These situations are listed in Chapter 20: Definitions of The TeXbook.

Rakishly speaking you can say that the only thing that might be "consumed" in some cases of gathering components of a <number> is <one optional space>`.


text:
# Find the intervals of concavity and the points of inflection of the following functions: $f(\theta)=\sin 2\theta$ in $(0 , \pi )$

Toolbox:
• Test for concavity Suppose $f$ is twice differentiable on an interval I.
• (i) If $f''(x) > 0$ for all $x \in I$, then the graph of is concerve upward (convex downward ) on I.
• (ii) If $f''(x) < 0$ fro all $x \in I$. then the graph of f is conceve downward ( convex upward) on I.
$f(\theta)=\sin 2 \theta, \theta \in (0, \pi)$
Step 1:
$f'(\theta)= 2\cos 2 \theta$
$f''(x)= -4\sin 2 \theta$
Step 2:
$f''(\theta)=0=> -4 \sin 2 \theta=0$
$=>\sin 2 \theta =0 \;or \; 2 \theta =0, \pi ; 2 \pi$
$=> \theta =0, \large\frac{\pi}{2}$$,\pi..... The value \theta= \large \frac{\pi}{2}$$\in (0,\pi)$
When $0 < \theta <\large\frac{\pi}{2}$$=> 0 <2 \pi <\pi f''(\theta)=-4 \sin 2 \theta < 0 (Since \sin 2 \theta > 0) (concaved downward) When \large\frac{\pi}{2}$$ < \theta < \pi =>\pi <2 \theta < \pi$
$f''(\theta)=-4 \sin 2 \theta > 0$ (Since $\sin 2 \theta < 0$) (concaved upward)
Step 3:
$\therefore f''(0)$ changes from concave downwards to concave upward at
$\theta= \large\frac{\pi}{2}$
$\theta=\large\frac{\pi}{2}$ corresponds to a point of inflection $f(\large\frac{\pi}{2})$$=2 \sin \pi=0 \therefore the point of inflection is at (\large\frac{\pi}{2}$$,0).$
answered Aug 5, 2013 by


text:
## Thursday, September 30, 2010

### 'Smart' electricity meter based on Efergy Elite monitor, part 1

Update (16 April 2011): I've got a follow-up post (part 2) which I describe a more efficient signal decoding technique using PIC interrupts and timers and allows for simple multitasking.

This series of posts describes how I hacked a low cost (€37 from Maplin) Efergy Elite electricity monitor to obtain real time power use data and to record and chart this data with the aim of understanding how electricty is used in the home and how to reduce consumption.

Describing the entire solution in one go is going to result in an excessively long post, so I'm going to break this into several parts over the next few weeks. In this post I'll describe the process involved in getting access to the raw data from the sensor unit.

The term 'smart' in the title is a misnomer. All I want to do here is record real time (at least several samples per minute) power use into a computer system. A true 'smart' meter will do telemetering, facilitate demand based tariffs, feed-in tariffs etc.

The problem with low cost energy monitors is you only get the current instantaneous usage and perhaps hourly, daily and weekly averages.

What I want is something that looks like this:

In this chart I can see my coffee machine (left on accidentally), the oven (not much I can do about that) and the fridge. From that chart I can figure the base load due to broadband router, WiFi access point, DVR etc and work out the average use of my fridge (by estimating the duty cycle and subtracting the power level while on from the base load).

So, how did I arrive at this chart?

I had an Efergy Elite electricity monitor which I received as a gift a few years ago. The product comprises a current transformer sensor which clips on to the utility meter outside. This transmits a power reading every 6 seconds over a 433MHz radio to the receiver unit indoors. The receiver has a LCD which displays instantaneous power use, and hourly, daily weekly averages (and other less interesting stuff like CO2 emissions). Photos of the sensor, receiver and internal electronics are here. There is no computer interface to tap into this data, so one needed to be hacked in.

There is a block diagram of my solution:

Looking at the receiver unit electronics, there is a test pad located near the radio part of the PCB. (The A72C01AUF chip to the bottom left of the test pad does all the radio heavy lifting both in the sensor and  receiver).

Looking at the signal at this point with an oscilloscope one can see there is burst of logic level (0/5V) signals every 6 seconds. This would seem to be the base band output from the radio.  An oscilloscope isn't the most useful tool for decoding digital IO if there are more than a few bits involved. I put a recently purchased Saleae 'Logic' (logic analyzer) on the task and this is what the signal looked like:

This signal didn't seem to make much sense. So I also simultaneously recorded the sensor's digital input line to the transmitter (pin 18 on the sensor's MCU – which by the way is a GSSP 22682X01 – a difficult chip to locate a datasheet for).

Thanks to the logic analyzer's big picture it seems there is an initial block of what looks like random noise preceding the actual data (I'm guessing this is just noise from the radio transmitter powering up).

Now looking more closely at the actual data:

The data is encoded using a digital equivalent of Frequency Shift Keying (FSK). A logic 0 is represented by three long pulses of 2ms duration in total (a low frequency) and a logic 1 by 4 shorter pulses totaling 2ms also (a high frequency).

Decoding this could be achieved with analog circuitry but this would involve a handful of (relatively) expensive passive components. Instead I chose a pure digital technique. I used a simple pattern matching algorithm to look for 3 successive long pulses or 4 successive short pulses and ignore anything else (code appended below).

From bits to bytes

So now I have a string of logic '0's and '1's. The next problem is to determine where the byte boundaries are.

Fortunately each data packet starts with the following 4 bytes: 0xAB 0xAB 0xAB 0x2D. This helps with synchronization. When looking for the start of data I shove the bits into a shift register until I get a byte value of 0xAB. I take all subsequent bits as groups of 8 to form the bytes of data. After the first 0xAB I ignore all subsequent byte values of 0xAB until I get a 0x2D. I am now at the start of the packet payload.

A typical packets look like this:
AB, AB, AB, 2D, 00, 0B, 5A, 40, 98, 00, 02, 00, 41
After some experimentation this is what I believe the packet consists of:

 0 Sync, always 0xAB 1 Sync, always 0xAB 2 Sync, always 0xAB 3 Sync end,  always 0x2D 4 ? always 0x00 5 Device address high byte (always 0x0D for my device) 6 Device address low byte  (always 0x5A for my device) 7 sbssaaaa where s = bits of sampling period, b = battery low indicator (0 = low) and aaaa is the upper 4 bits of sensor A. 0-00---- = 6s sampling, 1-01---- = 12s sampling, 0-10---- = 18s sampling. It's odd that ‘b’ is sandwiched in among s bits. 8 Least significant 8 bits of sensor input A (nearest MCU edge of board) 9 ? probably upper 4 bits of sensor B and C. 10 Least significant 8 bits of sensor input B (middle) 11 Least significant 8 bits of sensor input C 12 Checksum (arithmetical sum of bytes 4 - 11).

So in summary, each packet comprises a unique sensor address, the sampling rate, a battery low flag and 3 x 12 bit ADC values for each of the sensor's three inputs  (normal domestic power is single phase, so only one input is used. A three phase system would use all three inputs) and finally a checksum of the contents of the packet for data integrity.

I soldered a small (12F675) PIC MCU to a piece of veroboard (pictured right) and wrote a program to decode the baseband signal from the radio and convert that to actual data.

The PIC code used to decode the radio baseband signal is listed below (this excerpt is not the full PIC application – contact me if you need the lot):

The next problem is to figure how to translate these ADC values to Amps (and by multiplying by 230V to Watts) and how to feed this data to a database for recording and visualization. These will be covered in subsequent posts.

#include
#include "efergy_config.h"
#include "efergy_elite_decode.h"

byte getPulseWidth();
byte getBit ();
byte getByte ();
void resync();

byte t3,t4,tm;

/**
*
* Packet consists of a radio preamble of what seems
* like random noise for about 200ms followed by a quiet period
* of 100ms followed by data for 200ms. Encoding is a FSK type
* scheme.
* Logic 0 is represented by 3 cycles of "low" frequency square
* wave (period 8t per cycle).
* Logic 1 is represented by 4 cycles of "high" frequncy square
* wave (period 6t per cycle).
* Logic 0 and Logic 1 have the same period of 24t which is
* very approx 2ms [confirm]. Ie t ~= 83 microseconds.
*
*    +---+   +---+   +---+
*    |   |   |   |   |        = Logic 0
* +--|   +---+   +---+
*   +--+  +--+  +--+  +--+
*   |  |  |  |  |  |  |       = Logic 1
* +-+  +--+  +--+  +--+
*
* Decoding this not but looking at frequency, but looking
* for 3 long pulses in succession or 4 short pulses.
*
* Data has 3 bytes of synchronization: 0xAB, 0xAB, 0x2D.
* Look initially for 0xAB to get byte boundary synchronization.
* Then look for 0x2D to flag start of actual data.
*
* All inter-pulse calculation must be complete
* within 64x5 microseconds ie ~ 300 microseconds or 300 instructions @ 4MHz)
*/

byte decodeEfergy(unsigned char *buf) {
byte i,bitc=0, bytec=0, b=0;
byte t,pt;
//byte bufIndex=0;
unsigned int rt=0;

t3=0x1a; // short pulse duration of 3t
t4=0x23; // long pulse duration of 4t
tm = 0x1e; // mean of short and long pulse, ie 3.5t

resync:

// Look for sync bit sequence
// TODO: update: just look for AB followed by 2D.

// Get byte boundary sync
while (b != 0xAB) {
b = (b<<1) | getBit();
}
for (i = 0; i < 8; i++) {
b = (b<<1) | getBit();
}
if (b != 0xAB) {
goto resync;
}

for (i = 0; i < 8; i++) {
b = (b<<1) | getBit();
}
if (b != 0x2D) {
goto resync;
}

b=0;

// 9 bytes left to retrieve

do {
b = (b<<1) | getBit();

bitc++;
if (bitc == 8) {

buf[bytec++] = b;
b=bitc=0;
}
} while (bytec < 9);

// Calculate checksum
byte cs = 0;
for (i = 0; i < 8; i++) {
cs += buf[i];
}

// Display checksum fail if they don't match
if (cs != buf[8]) {
return 4;
}

return 0;
}

byte getByte () {
byte i,b;
for (i = 0; i < 8; i++) {
b = (b<<1) | getBit();
}
return b;
}

byte getBit () {
char t;
byte nshort=0,nlong=0;

while (1) {

t = getPulseWidth();

if ( t < tm) {
nshort++;
} else {
nlong++;
}

// Check for anomalous condition
if ( (nshort>0) && (nlong>0) ) {
nshort=nlong=0;
if ( t < tm) {
nshort=1;
} else {
nlong=1;
}
}

if (nshort == 4) {
return 1;
}
if (nlong == 3) {
return 0;
}
}
}

byte getPulseWidth() {
byte t=0;
CLRWDT();
}
t++;
}
return t;
}

void resync () {

byte pt,t;

restart_resync:

// Can have either 3t (short) or 4t (long) pulse. Get pulses of
// two different lengths. 3t is the short pulse, 4t is the long one.
pt = getPulseWidth();
do {
t = getPulseWidth()+2;
} while (t - pt < 4);
t -= 2;

if (pt < t) {
t3 = pt;
t4 = t;
} else {
t3 = t;
t4 = pt;
}

if (t4 < 0x1A) {
//TXREG='s';
goto restart_resync;
}
if (t4 > 0x30) {
//TXREG='l';
goto restart_resync;
}

// Didn't have to do this at the sensor, but receiver has a lot more incoming crud
if ( ((t4>>2)*3 ^ t3) & 0b11111100 ) {   // (t4/4)*3  ~= t3 (ignoring 2 LSB of precision)
//TXREG='?';
goto restart_resync;
}

// tm is mean of t3 and t4
tm = (t3+t4)>>1;

byte b=0;

// Look for sync bit sequence
while (b != 0xAB) {
b = (b<<1) | getBit();
}

}


text:
# عدد حقيقي فائق

في الرياضيات ، سيما في التحليل غير القياسي، الأعداد الحقيقية الفائقة hyperreal numbers أو الحقيقيات غير القياسية nonstandard reals (تمثل عادة ب *R) هي حقل مرتب يعتبر امتدادا لحقل الأعداد الحقيقية المرتب R يحقق مبدأ النقل transfer principle . هذا المبدأ يتيح لنا اعادة تفسير مقولات الدرجة الأولى حول R على أنها صحيحة أيضا في *R .

Infinitesimals (ε) and infinities (ω) on the hyperreal number line (1/ε = ω/1)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

## الاستخدام في التحليل

### الحسبان مع دوال جبرية

Informal notations for non-real quantities have historically appeared in calculus in two contexts: as infinitesimals, like dx, and as the symbol ∞, used, for example, in limits of integration of improper integrals.

As an example of the transfer principle, the statement that for any nonzero number x, 2x ≠ x, is true for the real numbers, and it is in the form required by the transfer principle, so it is also true for the hyperreal numbers. This shows that it is not possible to use a generic symbol such as ∞ for all the infinite quantities in the hyperreal system; infinite quantities differ in magnitude from other infinite quantities, and infinitesimals from other infinitesimals.

Similarly, the casual use of 1/0 = ∞ is invalid, since the transfer principle applies to the statement that division by zero is undefined. The rigorous counterpart of such a calculation would be that if ε is a non-zero infinitesimal, then 1/ε is infinite.

For any finite hyperreal number x, its standard part, st(x), is defined as the unique real number that differs from it only infinitesimally. The derivative of a function y(x) is defined not as dy/dx but as the standard part of the corresponding difference quotient.

For example, to find the derivative f′(x) of the function f(x) = x2, let dx be a non-zero infinitesimal. Then,

 ${\displaystyle f'(x)}$ ${\displaystyle =\operatorname {st} \left({\frac {f(x+dx)-f(x)}{dx}}\right)}$ ${\displaystyle =\operatorname {st} \left({\frac {x^{2}+2x\cdot dx+(dx)^{2}-x^{2}}{dx}}\right)}$ ${\displaystyle =\operatorname {st} \left({\frac {2x\cdot dx+(dx)^{2}}{dx}}\right)}$ ${\displaystyle =\operatorname {st} \left({\frac {2x\cdot dx}{dx}}+{\frac {(dx)^{2}}{dx}}\right)}$ ${\displaystyle =\operatorname {st} \left(2x+dx\right)}$ ${\displaystyle =2x}$

The use of the standard part in the definition of the derivative is a rigorous alternative to the traditional practice of neglecting the square[بحاجة لمصدر] of an infinitesimal quantity. Dual numbers are a number system based on this idea. After the third line of the differentiation above, the typical method from Newton through the 19th century would have been simply to discard the dx2 term. In the hyperreal system, dx2 ≠ 0, since dx is nonzero, and the transfer principle can be applied to the statement that the square of any nonzero number is nonzero. However, the quantity dx2 is infinitesimally small compared to dx; that is, the hyperreal system contains a hierarchy of infinitesimal quantities.

### التكامل

One way of defining a definite integral in the hyperreal system is as the standard part of an infinite sum on a hyperfinite lattice defined as a, a + dx, a + 2dx, ..., a + ndx, where dx is infinitesimal, n is an infinite hypernatural, and the lower and upper bounds of integration are a and b = a + n dx.[1]

1. ^ Keisler