text:
This repository has been archived by the owner. It is now read-only.

# Too many dependencies break the Windows file system #3697

Closed
opened this issue Jul 25, 2013 · 66 comments
Closed

# Too many dependencies break the Windows file system#3697

opened this issue Jul 25, 2013 · 66 comments
Labels
Milestone

### mcartoixa commented Jul 25, 2013

 On Windows, the file system itself cannot seem to cope very well with a too deeply nested node_modules directory... We just added bower to our dependencies, and now: the node_modules directory cannot be deleted from a batch command: DEL /S /Q /F node_modules fails. CLOC fails, even when told to ignore the node_modules directory completely. Git clean fails C:\PROGRA~2\Git\bin\git.exe clean -d -f -x. MSBuild item wildcards are not replaced anymore. We have tried to run npm dedupe to flatten our hierarchy but this is not enough. Effectively our Continuous Integration and delivery platform is broken for node.js/bower projects. Other people seem to have encountered the same kinds of problems on Windows. Any heads up? A more efficient dedupe command, perhaps? Using npm 1.3.2 on node.js 0.10.13. Reproduced on Windows 8 (x64) and Windows Server 2008 R2 (x64).

### dominictarr commented Aug 7, 2013

 oh, I've been experimenting with a new npm client, https://github.com/dominictarr/npmd you can do npmd install X --greedy and it installs everything directly into node_modules, only going deeper if there is a version conflict. This is all pretty experimental, and npmd is very young, so no promises, but it might fix your problem.

### ralt commented Aug 27, 2013

 This sounds like a huge issue. Is it a problem that npm cannot solve? Can we determine exactly what corrupts the filesystem? Is there no workaround? (Like switching an option on or off in ntfs.) The issue seems serious enough that time should be spent on it. Having no windows, I can't help, but from an external point of view, this looks serious.

### domenic commented Aug 27, 2013

 The file system is not corrupted. You just need to manipulate the files in question using the GUI or using UNC paths instead of the command line and traditional paths.

### mcartoixa commented Aug 27, 2013

 Maybe my wording is wrong, as could be my understanding of the problem. But the symptoms remain: the GUI is not a very practical solution in continuous integration scenarii. node.js tools like npm or grunt can fail too on our systems due to the mere presence of these folders (even on unrelated projects in different directories). I am not sure I could rewrite all our scripts to use UNC paths instead of traditional paths. @domenic Do you have more insight (maybe links) as to why UNC paths would work? I have once had to manually rename a whole hierarchy of folders to single letter names so that they could be deleted, even using the GUI...

### domenic commented Aug 27, 2013

 My impression was that libuv does this prefixing of "\\?\" transparently for you and so node programs should never run in to this problem. I have not verified however.

### mcartoixa commented Aug 28, 2013

 Here's an excerpt of a failing npm install output, due to filesystem related issues: unbuild isogeo-id@0.0.1 error : error rolling back Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob' error : error rolling back express-hbs@0.2.0 { [Error: ENOTEMPTY, rmdir 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob'] error : error rolling back errno: 53, error : error rolling back code: 'ENOTEMPTY', error : error rolling back path: 'D:\\ccnet\\projects\\Isogeo.App\\ContinuousBuild\\work\\node_modules\\isogeo-id\\node_modules\\express-hbs\\node_modules\\glob' } npm http GET https://registry.npmjs.org/i/-/i-0.3.2.tgz error : Error: ENOENT, lstat 'D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md' error : If you need help, you may report this log at: error : <http://github.com/isaacs/npm/issues> error : or email it to: error : <npm-@googlegroups.com> error : System Windows_NT 6.1.7601 error : command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" error : cwd D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\build error : node -v v0.10.13 error : npm -v 1.3.2 error : path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md error : fstream_path D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\node_modules\isogeo-id\node_modules\express-hbs\node_modules\glob\README.md error : fstream_type File error : fstream_class FileWriter error : code ENOENT error : errno 34 error : fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fstream\lib\writer.js:284:26 error : fstream_stack Object.oncomplete (fs.js:107:15) npm http GET https://registry.npmjs.org/esprima/-/esprima-1.0.3.tgz npm http 304 https://registry.npmjs.org/argparse  The same command would work on other OSs (this is Windows Server 2008 x64). A build on a Windows 7 x64 machine would fail at the grunt stage. Note that the folders described in the output are unlikely to be the cause of the trouble, but at this stage other deeply nested dependencies have already been fetched (based on bower on our case) that trigger all the kinds of strange behaviour described above.
mentioned this issue Sep 9, 2013

### JoseOneTime commented Dec 31, 2013

 Is there a resolution for this problem yet? I'm running into it on Windows 7.

### ghost23 commented Feb 5, 2014

 I am running into this, too. I can install almost no packages, always get errors like shown above. If i browse through the folder structure, i see huge nested 'node_modules' structures.

### clca commented Feb 6, 2014

 to repro the problem I did the following: create a temp directory ran npm install bower cd node_modules/bower npm install (at that point I could see the huge node_modules structure) I ran rmdir /s /q node_modules (from the temp directory) to delete the whole directory structure. It worked just fine on my Win8 machine.

### isaacs commented Feb 7, 2014

 It looks like this is related to a bug that's been fixed in later versions of rimraf and npm. Please update, and it should be fixed. While the Windows cmd program may have issues with long folder names, Node (and thus npm) always uses the UNC paths internally for fs operations, so it should work fine in npm. If you want to remove a folder, you can do npm install -g rimraf and then use the rimraf bin to do this. rimraf path\to\folder
closed this Feb 7, 2014

### calbertts commented Feb 11, 2014

 rimraf command works for me, thanks. Shit Windows!.

### mcartoixa commented Feb 11, 2014

 But what if I want to use CLOC or MSBuild item wildcards while the node_modules folder is still around?

### domenic commented Feb 11, 2014

 I suppose you can only hope those programs are as well designed as node and npm are, and can cope with paths longer than 256 characters!
mentioned this issue May 1, 2014

### neenhouse commented May 4, 2014

 I just wrote a node module "npm-flatten" that can do this. Check it out here: https://www.npmjs.org/package/npm-flatten. Source is here: https://github.com/neenhouse/npm-flatten
mentioned this issue Jun 1, 2014
mentioned this issue Jun 12, 2014

### ghost commented Jun 23, 2014

 An easy solution would be have an option to symlink dependencies directly into the top level node_modules folder. Say you had something like bower. The top level node_modules directory would have a folder structure looking something like: node_modules ├───abbrev │ └───1.0.5 ├───amdefine │ └───0.1.0 ├───ansi-styles │ └───1.0.0 ├───ansicolors │ └───0.2.1 ├───archy │ └───0.0.2 ├───argparse │ └───0.1.15 ├───array-filter │ └───0.0.1 ├───array-map │ └───0.0.0 ├───array-reduce │ └───0.0.0 ├───asn1 │ └───0.1.11 ├───assert-plus │ └───0.1.2 ├───async │ ├───0.2.10 │ └───0.9.0 ├───aws-sign │ └───0.3.0 ├───aws-sign2 │ └───0.5.0 ├───binary │ └───0.3.0 ├───block-stream │ └───0.0.7 ├───boom │ └───0.4.2 ├───bower │ └───1.3.4 ├───bower-config │ └───0.5.1 ├───bower-endpoint-parser │ └───0.2.1 ├───bower-json │ └───0.4.0 ├───bower-logger │ └───0.2.2 ├───bower-registry-client │ └───0.2.1 ├───buffers │ └───0.1.1 ├───cardinal │ └───0.4.4 ├───chainsaw │ └───0.1.0 ├───chalk │ └───0.4.0 ├───chmodr │ └───0.1.0 ├───cli-color │ └───0.2.3 ├───configstore │ ├───0.2.3 │ └───0.3.1 ├───cookie-jar │ └───0.3.0 ├───core-util-is │ └───1.0.1 ├───cryptiles │ └───0.2.2 ├───ctype │ └───0.5.2 ├───decompress-zip │ └───0.0.8 ├───deep-extend │ └───0.2.10 ├───es5-ext │ └───0.9.2 ├───esprima │ └───1.0.4 ├───event-emitter │ └───0.2.2 ├───forever-agent │ └───0.5.2 ├───form-data │ └───0.1.3 ├───fstream │ └───0.1.27 ├───fstream-ignore │ └───0.0.8 ├───glob │ └───3.2.11 ├───graceful-fs │ ├───2.0.3 │ └───3.0.2 ├───handlebars │ └───1.3.0 ├───has-color │ └───0.1.7 ├───hawk │ └───1.0.0 ├───hoek │ └───0.9.1 ├───http-signature │ └───0.10.0 ├───inherits │ └───2.0.1 ├───inquirer │ └───0.4.1 ├───insight │ └───0.3.1 ├───intersect │ └───0.0.3 ├───is-root │ └───0.1.0 ├───isarray │ └───0.0.1 ├───js-yaml │ └───3.0.2 ├───json-stringify-safe │ └───5.0.0 ├───jsonify │ └───0.0.0 ├───junk │ └───0.3.0 ├───lockfile │ └───0.4.2 ├───lodash │ └───2.4.1 ├───lodash.debounce │ └───2.4.1 ├───lodash.isfunction │ └───2.4.1 ├───lodash.isnative │ └───2.4.1 ├───lodash.isobject │ └───2.4.1 ├───lodash.now │ └───2.4.1 ├───lodash.objecttypes │ └───2.4.1 ├───lru-cache │ ├───2.3.1 │ └───2.5.0 ├───memoizee │ └───0.2.6 ├───minimatch │ └───0.3.0 ├───minimist │ ├───0.0.10 │ └───0.0.8 ├───mkdirp │ ├───0.3.5 │ └───0.5.0 ├───mkpath │ └───0.1.0 ├───mout │ └───0.9.1 ├───mute-stream │ └───0.0.4 ├───next-tick │ └───0.1.0 ├───nopt │ ├───1.0.10 │ └───2.2.1 ├───oauth-sign │ └───0.3.0 ├───object-assign │ ├───0.1.2 │ └───0.3.1 ├───opn │ └───0.1.2 ├───optimist │ ├───0.3.7 │ └───0.6.1 ├───osenv │ ├───0.0.3 │ └───0.1.0 ├───p-throttler │ └───0.0.1 ├───promptly │ └───0.2.0 ├───punycode │ └───1.2.4 ├───q │ ├───0.9.7 │ └───1.0.1 ├───qs │ └───0.6.6 ├───read │ └───1.0.5 ├───readable-stream │ └───1.1.13-1 ├───readline2 │ └───0.1.0 ├───redeyed │ └───0.4.4 ├───request │ ├───2.27.0 │ ├───2.34.0 │ └───2.36.0 ├───request-progress │ └───0.3.1 ├───request-replay │ └───0.2.0 ├───retry │ └───0.6.0 ├───semver │ ├───2.2.1 │ └───2.3.1 ├───shell-quote │ └───1.4.1 ├───sigmund │ └───1.0.0 ├───sntp │ └───0.2.4 ├───source-map │ └───0.1.34 ├───stringdecoder │ └───0.10.25-1 ├───stringify-object │ └───0.2.1 ├───strip-ansi │ └───0.1.1 ├───tar │ └───0.1.19 ├───throttleit │ └───0.0.2 ├───through │ └───2.3.4 ├───tmp │ └───0.0.23 ├───touch │ └───0.0.2 ├───tough-cookie │ └───0.12.1 ├───traverse │ └───0.3.9 ├───tunnel-agent │ ├───0.3.0 │ └───0.4.0 ├───uglify-js │ └───2.3.6 ├───underscore │ └───1.4.4 ├───underscore.string │ └───2.3.3 ├───update-notifier │ └───0.1.10 ├───uuid │ └───1.4.1 ├───which │ └───1.0.5 └───wordwrap └───0.0.2  Then each of the dependencies under node_modules for bower would be symlinked to these versions. Same would happen for dependencies of dependencies. At most you would only ever have a folder depth of 3 under node_modules (node_modules/bower/1.3.4/node_modules/) ├─┬ bower │ ├── abbrev (Symlink to ../../../abbrev/1.0.5) │ ├── archy (Symlink to ../../../archy/0.0.2) │ ├── bower-config (Symlink to ../../../bower-config/0.5.1) │ ├── bower-endpoint-parser (Symlink to ../../../bower-endpoint-parser/0.2.1) │ ├── bower-json (Symlink to ../../../bower-json/0.4.0) │ ├── bower-logger (Symlink to ../../../bower-logger/0.2.2) │ ├── bower-registry-client (Symlink to ../../../bower-registry-client/0.2.1) │ ├── cardinal (Symlink to ../../../cardinal/0.4.4) │ ├── chalk (Symlink to ../../../chalk/0.4.0) │ ├── chmodr (Symlink to ../../../chmodr/0.1.0) │ ├── decompress-zip (Symlink to ../../../decompress-zip/0.0.8) │ ├── fstream (Symlink to ../../../fstream/0.1.27) │ ├── fstream-ignore (Symlink to ../../../fstream-ignore/0.0.8) │ ├── glob (Symlink to ../../../glob/3.2.11) │ ├── graceful-fs (Symlink to ../../../graceful-fs/2.0.3) │ ├── handlebars (Symlink to ../../../handlebars/1.3.0) │ ├── inquirer (Symlink to ../../../inquirer/0.4.1) │ ├── insight (Symlink to ../../../insight/0.3.1) │ ├── is-root (Symlink to ../../../is-root/0.1.0) │ ├── junk (Symlink to ../../../junk/0.3.0) │ ├── lockfile (Symlink to ../../../lockfile/0.4.2) │ ├── lru-cache (Symlink to ../../../lru-cache/2.5.0) │ ├── mkdirp (Symlink to ../../../mkdirp/0.3.5) │ ├── mout (Symlink to ../../../mout/0.9.1) │ ├── nopt (Symlink to ../../../nopt/2.2.1) │ ├── opn (Symlink to ../../../opn/0.1.2) │ ├── osenv (Symlink to ../../../osenv/0.0.3) │ ├── p-throttler (Symlink to ../../../p-throttler/0.0.1) │ ├── promptly (Symlink to ../../../promptly/0.2.0) │ ├── q (Symlink to ../../../q/1.0.1) │ ├── request (Symlink to ../../../request/2.34.0) │ ├── request-progress (Symlink to ../../../request-progress/0.3.1) │ ├── retry (Symlink to ../../../retry/0.6.0) │ ├── semver (Symlink to ../../../semver/2.2.1) │ ├── shell-quote (Symlink to ../../../shell-quote/1.4.1) │ ├── stringify-object (Symlink to ../../../stringify-object/0.2.1) │ ├── tar (Symlink to ../../../tar/0.1.19) │ ├── tmp (Symlink to ../../../tmp/0.0.23) │ ├── update-notifier (Symlink to ../../../update-notifier/0.1.10) │ └── which (Symlink to ../../../which/1.0.5)  This would also help with deduplicating because you could easily just repoint the symlink to a module. This, from the programming point of view, wouldn't be difficult to implement. I'd happily submit a pull request, however it would be a question of whether or not it was considered the correct direction for the project manager as a whole.

### isaacs commented Jun 23, 2014

 So, like i said in my comment on joyent/node, the issue isn't that we haven't invented any flat reification strategies. There are plenty of those, and a lot of them work, even using symlinks. (One single global folder where everything lives, with symlinks all over the damn place, and swapping out the require.paths value, is not an ideal strategy, but there are many others.) We could, for example, have node_modules/.store// much like @Xeoncore is proposing. A problem you quickly run into is people doing require('.store/abbrev/1.2.3'), which, if you think people won't do, you have a short memory ;) We can avoid that by having the name be non-predictable, but then that kinda sucks for a lot of other reasons. One of the really nice things about nesting node_modules is that it will literally break half the time if you rely on it being in ./node_modules, so people don't do that for long So, let's say you have: node_modules +-- .store | +-- a | | +-- 1.2.3 | | +-- 2.5.4 | | | +-- b | +-- 1.2.3 | +-- 5.4.3 | +-- a -> .store/a/1.2.3 +-- node_modules +-- b -> ../.store/b/5.4.3  I guess you'd have to put the a/node_modules/b in .store/a/node_modules/b, and it's a symlink to a dir, so that's how it'd work anyway. But now you end up with a rats' nest pretty fast, because actually it's this: node_modules +-- .store | +-- a | | +-- 1.2.3 | | | +-- node_modules | | | +-- b -> ../../../.store/b/5.4.3 | | +-- 2.5.4 | | | +-- b | +-- 1.2.3 | +-- 5.4.3 | +-- a -> .store/a/1.2.3  Let's say b->c, and c->a@2.5.4, and a@2.5.4 depends on the other version of b. Updating a dep, then, is not a simple matter of replacing a single dir. you still have to build the tree, analyze the tree, reify the tree. We still have to figure out who needs what, and dedupe it as much as possible, and then analyze the changes, or else we're just trading deep trees for a giant .store dir. What's more, we MUST maintain the symlinks in each .store/*/node_modules dir, because otherwise they'll fall up to the top level node_modules folder, and potentially get an incompatible version! My point here is not to shoot down @Xeoncore's proposal, but merely to point out that coming up with the reification strategy is only a small part of the solution (albeit the creative fun part.) The other parts are much less fun: Figuring out the ways that people will abuse it, and how to make sure that they feel the pain of doing so as early as possible, when it is easy to correct. Working out all the ways that the strategy will fall over when there are deeply nested dep trees with lots of conflict management. Implementing the code that predictably guarantees dep version satisfaction in the craziest desert-island scenarios that we can imagine.

### ghost commented Jun 24, 2014

 @isaacs You make a good point with the user abuse situation. I wasn't suggesting a solution, more so trying to encourage discussion and development because in my opinion this is a legitimate and prevalent issue. To solve the issue with people directly using modules from the .store directory, is prohibit requiring of modules from it. That wouldn't be hard to do would it?

### pushpan999 commented Jun 25, 2014

 @isaacs Thanks a lot. I tried every possible option that I found in google, none of them worked. Finally rimraf worked :)

### StoneCypher commented Jul 3, 2014

 Fixing this in NPM isn't enough, because normal tools don't do things like this, so when you try to use node with other things, like the Windows Git package, these paths break them by proxy. Please stop pursuing the existing bad install strategy.

### ghost commented Jul 3, 2014

 @StoneCypher You shouldn't be committing your node_modules folder anyway.

### justrhysism commented Jul 14, 2014

 @Xeoncore You're right, we "shouldn't". But unfortunately, in the real-world, we need to. I've been bitten one too many times relying on package versions to handle this for me, and when a client calls for an update to an old project with old dependencies - and for whichever reason these dependencies aren't available, the project cannot be built and many, many hours are wasted trying to get a project to build just so a simply update can be done.

### ryanmargheriti commented Jul 24, 2014

 @Xeoncore Irrelevant.

### graphnode commented Sep 1, 2014

 Maybe I'm just showing my ignorance but couldn't we have a simple structure like: node_modules +-- a@1.2.3 | +-- a.js | +-- package.json +-- a@2.5.4 | +-- a.js | +-- package.json +-- b@1.2.3 | +-- b.js | +-- package.json +-- b@5.4.3 | +-- b.js | +-- package.json main.js package.json  Where, when a module requires another module, node.js tries to: Get the module from the local node_module folder; if not found: Get the module from the main node_module where the application started (like above); if not found: Get the module from the global node_module folder. Note: All of this with the specific version requested (from package.json) in mind. Any attempt to directly require a versioned module should throw a error (ex: require('a@2.5.4') or require('node_modules/a@2.5.4'): the versions used are in the package.json. No symbolic links are used and you can still use deep nesting if you wish (for module development?). What do you guys think?

### StoneCypher commented Sep 1, 2014

 Should work.

### bradleyflood commented Sep 2, 2014

 Sounds good @graphnode

### StoneCypher commented Jan 25, 2015

 @hopeseekr well, all other operating systems should conform to their expectations, you see, and if they've found an api that allows abuse, that means that all software globally should update to the thing they expect, and damn the consequences. this is especially easy since all software worldwide is actively being maintained, and nobody anywhere needs unsupported legacy software for anything. also what's a red book cd clearly that's much more straightforward than just having a sensible module deploy system in a single product that claims first class windows support, and didn't bother to learn how the filesystem worked

### StoneCypher commented Jan 25, 2015

 anything that doesn't work the way they expect is emphasis-bold broken, and should immediately conform to them

### othiym23 commented Jan 25, 2015

 @StoneCypher I know you're passionate about this, but I'd just like to point out that I've been doing all of my non-release process work on npm so far this year running Windows 8.1 as my sole OS (i.e. not in a vm), and I've already addressed several issues with Windows, as well as working with the io.js team to improve the Windows experience. In addition, npm@3 will be beta soon, and it builds a flatter dependency tree by default. I don't think the npm team can make any more plain that we do care about Windows and are committed to improving the experience of npm on it.

### StoneCypher commented Jan 25, 2015

 The commentary mostly regards community positions and tone. As before, I'm engaging in someone else's pre-existing commentary theretowards. I like node. A lot. I continue to use it. I am slowly becoming productive in it. However, every time Windows people point out where Node is doing severely broken things unnecessarily, like abusing the internationalization api's widechar support to stuff bytes where the API explicitly states they're not permitted to be, we are met with claims that because we do things differently, we suck and are broken, and that 40 years of other software's stability should be violated so that a marginally more sophisticated layout algorithm need not be deployed. We're fine with "this is pretty hard to do because it turns out that it involves rewriting a bunch of the core." ljharb told me about that the other day. I didn't get a clear context on the problem, and to be frank it seems super weird, but if that's the case, then fine. But that's not what we got. We got mocking and taunting and sophistry that didn't make sense. This community has an admirable focus on choice of wording to prevent subtle hostility from excluding people. The gender pronoun debate, though I think handled poorly, was on the right topic. You do realize that more than ninety percent of developers are Windows goons, right? Have you ever wondered why you have so few of them? I am glad that you choose to stamp out the exclusory behaviors that you can see, so as to make a more inclusive environment where you can get the greatest participation. I just wish that you saw more of them. People hang lanterns on them all the time. Hopeseeker's a fun dude. I met him in the PHP community. Don't ignore him just because you see my name.

### am11 commented Jan 25, 2015

 Guys what I have learned from IRC (freenode -> nodejs@freenode channel) is that this issue will be fixed in npm v3 as @othiym23 mentioned. Also @iarna indicated above that she was working on it. TBVH, this is no straightforward workaround for this 256 char limitation of NTFS (not very new after all). Not sure what is the situation with ReFS. Nonetheless, I also sent a feedback for this on Windows 10 preview, _hopefully others will do the same_ in order to make them realize that this limitation needs to be revoked and/or reworked. IMO, as a whole npm team has done much better job compared to other package managers (Nuget Packages, Ruby Gems etc.). Well the reluctance to Windows in any non-MSFT technology community is very common. Although, some people are quite transparent in delivering their concern (for instance the guy who wrote dietlibc (which I was using to make node-sass binary cross-Linux-distro compatible) has to say something in his no windows passage), but on that stance they forget that others (MacWhack) are doing similar stuff (making revenue -- limiting freedom). Since WindOS being non-pure-Unix-like, it will keep getting this special treatment which is not something really new / surprising. :) On the flipside, see the work from neovim and clang folks, how hard they had to struggle to make their software compatible with Windows, when VC team was not making it easier for masses up until very recently (C99, C11, C++11, C++14 STL and lang features and since so forth).

### iarna commented Jan 25, 2015

 @StoneCypher When you complain here about "you all" not taking windows seriously, and "here" is an issue on npm, it's hard to not take that as "npm is not taking windows issues seriously." We're aware that we've got some work to build back trust for Windows users, but I'm hopeful that 2015 will be a year that will change that perception. As far as fixing issues with nodejs, you might try raising the issues again over on the iojs issue board (if you haven't already), as it's possible they'll be more receptive. @am11 That's right: npm@3's dependency resolution algorithm makes the node_modules folder as flat as possible (that is, if there are no conflicts, it puts all requirements at the top level. This results in a implicitly deduped and very shallow tree. Outside of pathological cases, I anticipate that this will solve the "256 char limit" problem by never producing folders nested deeply enough to hit it.

### hopeseekr commented Jan 26, 2015

 @iarna That's great news! And it's a complete reversal from the first comments about this issue, that went something like, "Haha! You run Windows? You Plebe! Suck it!"

### StoneCypher commented Jan 26, 2015

 @iarna The io.js people have been great; I'm a semi-regular in their channel.

### droppedoncaprica commented Jan 26, 2015

 @StoneCypher As a non-npm or node/io developer just lurking the issue trackers, you've always seemed very combative on the subject of Windows integration, which is why I assume most people aren't taking you seriously on it. Not trying to poke the bear here; I understand you're frustrated. But I do think the developres are moving towards a solution here.

### StoneCypher commented Jan 26, 2015

 @droppedoncaprica I understand how telling people how it feels to be treated this way must seem combative, yes.

### johnnyreilly commented Jan 28, 2015

 @iarna - great to hear that npm@3 is likely to improve life in Windows. Is there any thought as to when that might ship?
mentioned this issue Feb 8, 2015
mentioned this issue Feb 17, 2015
mentioned this issue Feb 26, 2015

### nurbek-ab commented Mar 25, 2015

 @StoneCypher thanks a lot! subst solution provided by you works like a charm. All other solutions suggested here didn't work for me but by simply mounting node_modules folder to Z drive I was able to remove the whole folder structure.
mentioned this issue Mar 29, 2015
mentioned this issue Mar 30, 2015
mentioned this issue Apr 1, 2015
mentioned this issue Apr 1, 2015
mentioned this issue Apr 14, 2015

### DullReferenceException commented Apr 30, 2015

 Is this still being tracked? I see it's been closed but not resolved. My perception when a non-resolved issue is closed is that it's being ignored, which I really hope is not the case.

### kevva commented Apr 30, 2015

 It's being fixed in npm@3.

### deostroll commented May 4, 2015

 You can uninstall node modules in that folder serially to solve the problem. http://stackoverflow.com/a/29915329/145682
mentioned this issue May 16, 2015

### chrispappas commented May 28, 2015

 Found this suggestion for using robocopy with an empty directory and node_modules, using the purge switch: http://stackoverflow.com/a/3320317 Paraphrased TL;DR: robocopy empty_dir node_modules /purge I wasn't able to install rimraf because of a borked up npm directory... hoping I can get everything back working again. Hope this helps someone!

### crertel commented Jun 2, 2015

 So, what is the solution in npm@3 going to be? And how far away is that, exactly? I'm rather blocked on this at work, and honestly I'm really getting a bit sick of dealing with the Node ecosystem's quirks. I don't want to clutter up this thread with continued kvetching, but I'm really a bit puzzled that a problem well-solved by, say, Bundler and Hex is somehow still causing grief. (And unfortunately, yes, some of us are stuck with Windows deployments--and no, we'd rather not be, but them's the breaks.)

### iarna commented Jun 23, 2015

 npm@3 makes the install "maximally flat" by hoisting everything it can to the top level node_modules. This means nesting only occurs on conflicts and as such, trees should never get very deep. As such, the windows path length limitation shouldn't be run into. Timeline-wise, we're looking to do the first public release this week.

### hopeseekr commented Jun 24, 2015

 Thank God[dess]!! After the release, NodeJS can announce it once again supports Windows operating systems! I've been battling this issue for... get this!! 2 years and 4 months! For such a critical issue for so many of my clients, I am very happy something is finally being done. Because, frankly, I've had to create a pure PHP equivalent of npm, Gulp and Grunt that doesn't have the damn nested problem. And it works a lot better, too.

### iarna commented Jun 24, 2015

 I'm locking comments here as they're no longer constructive. Anything new issues around the things discussed here should be opened as new issues.
locked and limited conversation to collaborators Jun 24, 2015


text:
0
Research Papers: Heat Transfer Enhancement

# Computational Analysis of Surface Curvature Effect on Mist Film-Cooling Performance

[+] Author and Article Information
Xianchang Li1

Department of Mechanical Engineering, Lamar University, Beaumont, TX 77710xianchang.li@lamar.edu

Ting Wang

Energy Conversion & Conservation Center, University of New Orleans, New Orleans, LA 70148-2220

1

Corresponding author.

J. Heat Transfer 130(12), 121901 (Sep 15, 2008) (10 pages) doi:10.1115/1.2970071 History: Received August 29, 2007; Revised April 16, 2008; Published September 15, 2008

## Abstract

Air-film cooling has been widely employed to cool gas turbine hot components, such as combustor liners, combustor transition pieces, turbine vanes, and blades. Studies with flat surfaces show that significant enhancement of air-film cooling can be achieved by injecting water droplets with diameters of $5–10 μm$ into the coolant airflow. The mist/air-film cooling on curved surfaces needs to be studied further. Numerical simulation is adopted to investigate the curvature effect on mist/air-film cooling, specifically the film cooling near the leading edge and on the curved surfaces. Water droplets are injected as dispersed phase into the coolant air and thus exchange mass, momentum, and energy with the airflow. Simulations are conducted for both 2D and 3D settings at low laboratory and high operating conditions. With a nominal blowing ratio of 1.33, air-only adiabatic film-cooling effectiveness on the curved surface is lower than on a flat surface. The concave (pressure) surface has a better cooling effectiveness than the convex (suction) surface, and the leading-edge film cooling has the lowest performance due to the main flow impinging against the coolant injection. By adding 2% (weight) mist, film-cooling effectiveness can be enhanced approximately 40% at the leading edge, 60% on the concave surface, and 30% on the convex surface. The leading edge film cooling can be significantly affected by changing of the incident angle due to startup or part-load operation. The film cooling coverage could switch from the suction side to the pressure side and leave the surface of the other part unprotected by the cooling film. Under real gas turbine operating conditions at high temperature, pressure, and velocity, mist-cooling enhancement could reach up to 20% and provide a wall cooling of approximately 180 K.

<>

## Figures

Figure 2

Meshes of partial domain and details near injection holes (with grid adaptation)

Figure 3

Flow pattern and temperature distribution

Figure 4

Comparison of air-only leading-edge adiabatic cooling effectiveness with that on a flat surface with the same nominal blowing ratio of 1.33

Figure 5

Droplet trajectories versus jet flow pathlines for leading-edge mist cooling

Figure 6

Enhancement of adiabatic cooling effectiveness by injecting mist with the nominal blowing ratio of 1.33

Figure 7

Effects of the droplet size and mist concentration on film-cooling enhancement at the leading edge

Figure 8

Effect of the main flow incident angle on flow and temperature distribution close to the leading edge

Figure 9

Effect of the main flow incident angle on adiabatic film-cooling effectiveness near the leading edge

Figure 1

Schematic of film cooling and computational domain: (a) film-cooling concept, (b) 2D cases, and (c) 3D cases

Figure 10

Effect of elevated operating conditions on mist film-cooling performance of the turbine blade leading edge

Figure 11

Flow pattern and temperature distribution of film cooling under the curvature effect on the pressure and suction sides

Figure 12

Comparison of air-only film-cooling effectiveness on flat and curved surfaces with a nominal blowing ratio of 1.33

Figure 13

Enhancement of film cooling with mist injection on concave and convex surfaces

Figure 14

2D droplet trajectories and vapor concentration in mist film cooling with concave and convex surfaces

Figure 15

Temperature distribution of air-film cooling on 3D leading-edge surface

Figure 16

Adiabatic cooling effectiveness of 3D leading-edge film cooling and its enhancement with mist injection

## Errata

Some tools below are only available to our subscribers or users with an online account.

### Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related Proceedings Articles
Related eBook Content
Topic Collections

format: plain


text:
# Infinite series by integration by parts

1. Aug 4, 2009

### disregardthat

Hi, I wonder if this hypothesis is true:

Let $$f_n$$ be an arbitrarily chosen n'th anti-derivative of the function $$f_0$$. Similarly, let $$g_n$$ be the n'th derivative of the function $$g_0$$.

Now, $$\int^b_a f_0 g_0 \rm{d}x=[f_1g_0]^b_a-\int^b_a f_1g_1 \rm{d}x=[f_1g_0-f_2g_1+...]^b_a+(-1)^n \int^b_a f_{n+1}g_n \rm{d}x$$.

hypothesis:

If $$\lim_{n \to \infty} f_{n+1}g_n =0$$ for all continous intervals of and never diverges. Then

$$\int^b_a f_0 g_0 \rm{d}x = [\sum^{\infty}_{n=0} (-1)^n f_{n+1}g_n]^b_a$$

This seems intuitively correct, but I wonder how to prove it.

Last edited: Aug 4, 2009
2. Aug 4, 2009

### Jame

I'm thinking Mean Value Theorem. The integral equals $$\epsilon(b-a)(f_{n+1}g_n)|_{x=c}$$ for some c in the interval [a, b] and $$0 \leq \epsilon \leq 1$$ which approaches zero.

3. Aug 4, 2009

### disregardthat

this was double post, sorry

Last edited: Aug 4, 2009
4. Aug 4, 2009

### disregardthat

Excellent, thank you.

We have by induction that

$$\int^b_a f_0g_0 \rm{d}x = [\sum^{n-1}_{k=0} (-1)^kf_{k+1}g_{k}]^b_a+(-1)^n\int^b_af_{n+1}g_n \rm{d}x=[\sum^{n-1}_{k=0} (-1)^kf_{k+1}g_{k}]^b_a+(b-a)(f_{n+1} \circ g_n)(t)$$

For some $$t \in [a,b]$$, and any non-negative integer n.

However, $$\lim_{n \to \infty} (f_{n+1} \circ g_{n} )(t)=0$$ is given, so

$$\int^b_a f_0g_0 \rm{d}x = \lim_{n \to \infty} \int^b_a f_0g_0 \rm{d}x= \lim_{n \to \infty} [\sum^{n-1}_{k=0} (-1)^kf_{k+1}g_{k}]^b_a+(-1)^n\int^b_af_{n+1}g_n \rm{d}x=\lim_{n \to \infty}[\sum^{n-1}_{k=0} (-1)^kf_{k+1}g_{k}]^b_a+(b-a)(f_{n+1} \circ g_n)(t)=[\sum^{\infty}_{k=0} (-1)^kf_{k+1}g_{k}]^b_a$$

But this is not so obvious if $$\lim_{n \to \infty} (f_{n+1} \circ g_{n} )(t)$$ not always equal 0 i.e. is finite for discrete values of x, or if either of the limits are infinite. Can someone help me there?

Perhaps if the limits are infinite, we can let n tend towards infinity at a rate which make $$f_{n+1}g_n$$ dominate a limit, say b i.e. so $$\lim_{b,n \to \infty} b \cdot f_{n+1}g_n = 0$$ Can we choose it to be like that?

Last edited: Aug 4, 2009


text:
# Matrix | Row Matrix | Column Matrix | Square Matrix | Identity Matrix

Matrix is an arrangement of element in rows and column in a rectangular array.

For Example :-

A =  ${ \begin{bmatrix} 1 &\quad 4 &\quad -2\\ 3 &\quad 7 &\quad 3\\2 &\quad-1 &\quad7 \end{bmatrix} }_{ 3\times 3 }$

B = ${ \begin{bmatrix} 6 & \quad 1 & \quad -2 \\ 4 & \quad 2 & \quad 5 \end{bmatrix} }_{ 2\times 3 }$

Row Matrix
Row Matrix is a matrix containing a single row.

For Example :-

A = ${ \begin{bmatrix} 2 & \quad -5 & \quad 1 \end{bmatrix} }_{ 1\times 3 }$

B = ${ \begin{bmatrix} 4 & \quad 3 & \quad -6 \end{bmatrix} }_{ 1\times 3 }$

Column Matrix
Column Matrix is a matrix containing a single coulumn.

For Example :-

A = ${ \begin{bmatrix} 1 \\ 3 \\ 5 \end{bmatrix} }_{ 3\times 1 }$

A = ${ \begin{bmatrix} 7 \\ 2 \\ 4 \end{bmatrix} }_{ 3\times 1 }$

Square Matrix
Square Matrix is a matrix having equal numbers of rows and column.

For Example :-

A = ${ \begin{bmatrix} 2 &\quad -1 \\ 3 &\quad 5 \end{bmatrix} }_{ 2\times 2 }$

B = ${ \begin{bmatrix} 1 &\quad 3 &\quad -2\\ 4 &\quad 6 &\quad 3\\2 &\quad-1 &\quad7 \end{bmatrix} }_{ 3\times 3 }$

Identity Matrix
If the diagonal element of a square matrix are 1 and other element are 0, then the matrix is Known as Identity Matrix.

For Example :-

A = ${ \begin{bmatrix} 1 &\quad 0 \\ 0 &\quad 1 \end{bmatrix} }_{ 2\times 2 }$

B = ${ \begin{bmatrix} 1 &\quad 0 &\quad 0\\ 0 &\quad 1 &\quad 0\\0 &\quad0 &\quad1 \end{bmatrix} }_{ 3\times 3 }$


text:
The ratio of the sums of m and n terms of an A.P. is m2: n2. Show that the ratio of mth and nth term is (2m – 1): (2n – 1).

Asked by Abhisek | 1 year ago |  72

##### Solution :-

Let’s consider that a and b to be the first term and the common difference of the A.P. respectively.

Then from the question, we have

$$\dfrac{s_m}{s_n}=\dfrac{m^2}{n^2}$$

$$s_n=\dfrac{\dfrac{m}{2}[2a+(m-1)d]}{\dfrac{n}{2}[2a+(n-1)d]}=\dfrac{m^2}{n^2}$$

$$=\dfrac{2a+(m-1)d}{2a+(m+1)d}$$

= n[2a+(m−1)d]=m[2a+(n−1)d]

⇒2an+mnd−nd+2am+mnd−nd

= md−nd=2am−2an

= (m−n)d=2a(m−n)

=d=2a

Now, the ratio of mth and nth terms is

$$\dfrac{m^2}{n^2} =\dfrac{a+(m-1)d}{a+(m+1)d}$$

$$\dfrac{a+(m-1)2a}{a+(m+1)2a}$$

$$\dfrac{a(1+2m-2)}{a(1+2n-2)}$$

$$\dfrac{2m-1}{2n-1}$$

Therefore, the ratio of mth and nth term is (2m -1) : (2n -1)

Answered by Pragya Singh | 1 year ago

### Related Questions

#### Construct a quadratic in x such that A.M. of its roots is A and G.M. is G.

Construct a quadratic in x such that A.M. of its roots is A and G.M. is G.

#### Find the two numbers whose A.M. is 25 and GM is 20.

Find the two numbers whose A.M. is 25 and GM is 20.

#### If a is the G.M. of 2 and 1/4 find a.

If a is the G.M. of 2 and $$\dfrac{1}{4}$$ find a.

#### Find the geometric means of the following pairs of numbers

Find the geometric means of the following pairs of numbers:

(i) 2 and 8

(ii) a3b and ab3

(iii) –8 and –2

Insert 5 geometric means between $$\dfrac{32}{9}$$ and $$\dfrac{81}{2}$$.


text:
Connexions

You are here: Home » Content » Connexions Tutorial and Reference » Using Basic CNXML in Edit-In-Place

Lenses

What is a lens?

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags?

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

Endorsed by (What does "Endorsed by" mean?)

This content has been endorsed by the organizations listed. Click each link for a list of all content endorsed by the organization.
• HPC Open Edu Cup

This module is included inLens: High Performance Computing Open Education Cup 2008-2009
By: Ken Kennedy Institute for Information TechnologyAs a part of collection: "2008-'09 Open Education Cup: High Performance Computing"

Click the "HPC Open Edu Cup" link to see all content they endorse.

Click the tag icon to display tags associated with this content.

Affiliated with (What does "Affiliated with" mean?)

This content is either by members of the organizations listed or about topics related to the organizations listed. Click each link to see a list of all content affiliated with the organization.
• CNX Documentation

This module and collection are included inLens: Connexions Documentation
By: Connexions

"The canonical how-to guide to using Connexions."

Click the "CNX Documentation" link to see all content affiliated with them.

Click the tag icon to display tags associated with this content.

Also in these lenses

• eScience, eResearch and Computational Problem Solving

This module is included inLens: eScience, eResearch and Computational Problem Solving
By: Jan E. OdegardAs a part of collection: "2008-'09 Open Education Cup: High Performance Computing"

Click the "eScience, eResearch and Computational Problem Solving" link to see all content selected in this lens.

• OER/LOR Connexions T

This collection is included inLens: OER/LOR Connexions Training
By: Connexions

"This collection has the basic training for authoring modules (chapters/sections) and collections (textbooks/courses etc)."

Click the "OER/LOR Connexions T" link to see all content selected in this lens.

Click the tag icon to display tags associated with this content.

Recently Viewed

This feature requires Javascript to be enabled.

Tags

(What is a tag?)

These tags come from the endorsement, affiliation, and other lenses that include this content.

Inside Collection (Course):

Using Basic CNXML in Edit-In-Place

Module by: Elizabeth Gregory, Connexions. E-mail the authors

Summary: This document introduces simple CNXML tags that are easy to use in Edit-in-Place.

This module also contains information derived from The Advanced CNXML by Ricardo Radaelli-Sanchez.

Starting with CNXML

To create the bare bones of content in Connexions, the author interface provides a variety of creation tools: the Document Importer, Edit-In-Place, and even a full-source editor. However, a basic knowledge of our markup language can help make small edits into tremendous enhancements to your material!

Connexions uses the Connexions Markup Language (CNXML) as its primary language for marking up and storing documents. CNXML is lightweight XML for marking up educational content. Unlike the well-known HTML, the goal of CNXML is to convey the content of the material and not a particular presentation. For example, say you have the following sentence: I like cupcakes very much. However, you feel that you enthusiasm for cupcakes has not been fully expressed. In HTML, you would use bold, underline, italic, etc.; in CNXML, you would use the emphasis tag.

Inline Tags

Inline tags, such as emphasis, are used to embed content and functionality inside the structural tags, such as paragraphs. Some of the more commonly used tags are discussed below.

Emphasis

As mentioned previously, the emphasis tag is used to accent certain text. Note that this refers to semantic emphasis and not a typeface. Different stylesheets can render emphasis with different typefaces.

Example 1

<para id='intro'>
Gardenias are my absolute <emphasis>favorite</emphasis>
flower. Their petals are soft, and their bloom has an
absolutely <emphasis>heavenly</emphasis> scent.
</para>

The above markup will display as:

Gardenias are my absolute favorite flower. Their petals are soft, and their bloom has an absolutely heavenly scent.

Term

The term tag is used to mark words or phrases which are being defined. However, its use is confined to either a para or definition tag. The term tag has one optional attribute: URL - a URL specifying the source or definition of the term.

Example 2

<para id='gardenia'>
<term url="http://en.wikipedia.org/wiki/Gardenia">Gardenias
</term> can be tricky to maintain.  The soil around the
roots of a <term>gardenia</term> must remain moist always,
but too much water can damage the plant.  Also,
<term>gardenias</term> enjoy the sun, but if the
<emphasis>foliage</emphasis> gets wet to bring the gardenia
</para>

The above markup will display as:

Gardenias can be tricky to maintain. The soil around the roots of a gardenia must remain moist always, but too much water can damage the plant. Also, gardenias enjoy the sun, but if the foliage gets wet to bring the gardenia into the shade.

Note

The note tag creates an "out of line" note to the reader. The type of note is specified by an optional type attribute. If a type is not specified, the default is Note. The type attribute can contain any of the following values:

• note
• aside
• warning
• tip
• important

Example 3

<para id='pollen'>
Receiving flowers is, on the whole, a wonderful thing.
However, sometimes pollen from the flowers can cause
problems.  In particular, the clean up of a bit of
pollen can be tricky.  <note type="Important">Do not
use water when cleaning up pollen!  This can lead to
counter-top and clothing stains!</note>  Your best bet
is to use a dry method of cleaning with a paper-towel.
</para>

The above markup will display as:

Receiving flowers is, on the whole, a wonderful thing. However, sometimes pollen from the flowers can cause problems. In particular, the clean up of a bit of pollen can be tricky.

Important:
Do not use water when cleaning up pollen! This can lead to counter-top and clothing stains!
Your best bet is to use a dry method of cleaning, with a paper-towel, for example.

The link tag is the tag in CNXML used for linking to other Connexions modules or collections as well as external links.

• strength: The Strength attribute can contain the value 1, 2, or 3 specifying the relevance of the link.
• window: The Window attribute determines the manner in which the link location will be opened. It can contain the values "Replace" or "New". "Replace" will result in the link location opening in the current window replacing the page with the link. "New" will result in the link location opening in a new browser window.
• url: The URL attribute can contain the web address of the link you wish to reference.
• document: The Document attribute is used to reference the ids of other Connexions modules or Collections.
• target-id: The Target-id attribute is used to reference the ids of specific elements within Connexions modules.
• resource:
• version: The Version attribute is used to reference a specific version of a Connexions module or collection.

The target and document attributes can be used together or alone. If both are used then you will link to a particular tag in another document. If only document is used, you will link to another document. If only target is used, you will link to a particular tag within the current document.

Cite

The cite tag is used to refer to non-electronic materials within a document, and primarily contains the title, the author, and/or a page number of a work.

Example 4: Cite Example

Finally, a good resource is the Garden Lover's
Cookbook -- William M. Rice; Paperback.

The above markup will display as:

Finally, a good resource is the Garden Lover's Cookbook -- William M. Rice; Paperback.

Quote

The quote tag is used to denote that some text directly quotes another source. The quote tag has a display attribute which denotes whether the quote is inline or block.

Example 5: Quote Example

<para id='plantquote'>
Every plant needs a different amount of water in order to
grow well. <quote display="inline">"If you water each plant the same, you
will always water too much and too little."</quote>  Also,
remember the words of Lou Erickson:
<quote id="quote_example" display='block'>"Gardening requires lots of water -
most of it in the form of perspiration."</quote>
</para>
Every plant needs a different amount of water in order to grow well. “"If you water each plant the same, you will always water too much and too little."” Also, remember the words of Lou Erickson:
"Gardening requires lots of water - most of it in the form of perspiration."

Foreign

The foreign tag is used to denote that a word or phrase foreign to the language of the document is being used.

Example 6: Foreign Example

<para id='plantquote2'>
All flowers have a scientific name, often derived from
Latin.  <foreign>Gardenia augusta</foreign> is the name
of a type of gardenia found in Japan.
</para>
All flowers have a scientific name, often derived from latin. Gardenia augusta is the name of a type of gardenia found in Japan.

Code

The code tag is used to insert example computer output/input as either inline text within a paragraph or as a block of text. The code tag has a display attribute with two possible values:

• inline (default) - used to specify code that is inline.
• block - used to specify code that should be in a separate block of text.

Example 7: Inline Code Example

For now, take a look at what the inline code looks like:

<para id='copy'>
In a unix terminal the command to copy a file is
cp original copy.
</para>

In a unix terminal the command to copy a file is cp original copy

You will see more about code blocks in Advanced CNXML using Edit-In-Place.

Content actions

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Collection to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags?

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks

Module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags?

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks


text:
# Compare similarities between two data frames using more than one column in each data frame

This work started by comparing two columns in each data set in pandas.
Previous research:here
A lot of results online show how to compare 2 data frames with 1 column
I'm trying to learn how to compare and extract similarities between two data frames (same & different sizes if possible) using more than 1 column in pandas.

sample input:

df1=pd.DataFrame([[1,None],[1,None,],[1,None],[1,'item_a'],[2,'item_a'],[2,'item_b'],[2,'item_f'],[3,'item_e'],[3,'item_e'],[3,'item_g'],[3,'item_h']],columns=['id','A'])
df2=pd.DataFrame([[1,'item_a'],[1,'item_b'],[1,'item_c'],[1,'item_d'],[2,'item_a'],[2,'item_b'],[2,'item_c'],[2,'item_d'],[3,'item_e'],[3,'item_f'],[3,'item_g'],[3,'item_h']],columns=['id','A'])

 df1
id  A
0   1   None
1   1   None
2   1   None
3   1   item_a
4   2   item_a
5   2   item_b
6   2   item_f
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h

df2
id  A
0   1   item_a
1   1   item_b
2   1   item_c
3   1   item_d
4   2   item_a
5   2   item_b
6   2   item_c
7   2   item_d
8   3   item_e
9   3   item_f
10  3   item_g
11  3   item_h

What I've tried so far:

   1: df1[df1.A.isin(df2.A) & df1.id.isin(df2.id)]

2: df1[   df1[['id', 'A']].isin(df2[['id', 'A']])  ]

The output I got for 1 is close to what I desire:

    id  A
3   1   item_a
4   2   item_a
5   2   item_b
6   2   item_f #this specific row is not desired in the output
7   3   item_e
8   3   item_e #this specific row was raised due to a duplicate in df1. It's permitted to show duplicates. Duplicates values are allowed in df1 but not df2.
9   3   item_g
10  3   item_h

Desired output:

    id  A
3   1   item_a
4   2   item_a
5   2   item_b
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h

What's not shown: Two data frames have 2500+ rows. df1 can have the same items associated with an id. No duplicate items for an id in df2.
My 2nd try 2: df1[ df1[['id', 'A']].isin(df2[['id', 'A']]) ] is definitely the wrong approach as its matching row and column in df1 to row and column df2 (This output is similar to equals(), I get values from df1 instead of True and NaN instead of False)
Any code, links, suggestions are appreciated.


text:
# Scalable models workshop recap

[ This blog post is collaboratively written by Evan and Memming ]
The Scalable Models workshop was a remarkable success! It attracted a huge crowd from the wee morning hours till the 7:30 pm close of the day. We attracted so much attention that we had to relocate from our original (tiny) allotted room (Superior A) to a (huge) lobby area (Golden Cliff). The talks offered both philosophical and methodological perspectives, reflecting diverse viewpoints on and approaches to high-dimensional neural data. Many of the discussions continued the next day in our sister workshop. Here we summarize each talk:

### Konrad Körding – Big datasets of spike data: why it is coming and why it is useful

Konrad started off the workshop by posing some philosophical questions about how big data might change the way we do science. He argued that neuroscience is rife with theories (for instance, how uncertainty is represented in the brain) that, while supported by some data, are plausible only for particular areas or computations. Konrad argued that this may simply be the state of affairs for the brain, where complexity and heterogeneity are the rule. Rather than reject such theories as incomplete, perhaps we should be humble about our theories. Broad, overarching principles may only ever explain a small percentage of the overall variance in the brain.

In the second part of his talk, Konrad discussed his recent work in using Fisher information to build a unifying theory of large-scale neural recording techniques [Cybulski et al. 2014]. For example, it can be used to find the best distribution of electrodes in neural tissue.

• Cybulski, T. R., Glaser, J. I., Marblestone, A. H., Zamft, B. M., Boyden, E. S., Church, G. M., and Kording, K. P. (2014). Spatial information in Large-Scale neural recordings.

### Jeremy Freeman – Distributed computing for large-scale neuroscience

Neuroscience will need new software tools to handle the very large-scale data produced by large-scale experiments (in both time and # neurons). Luckily, as Jeremy Freeman observed, neuroscience can lean on many tools already developed by the tech industry to deal with “big data”. One such tool is Map-Reduce: a framework for massive parallelization often used in web search and elsewhere. Jeremey introduced a software framework called Spark that provides a more abstract interface more suitable for iterative calculations that are often required in neuroscience. Jeremy advocates Spark for the use in large-scale data analysis, and is actively developing a software package called Thunder that implements many common data analysis tools such as PCA, ICA, and regression on top of Spark. For a 0.1TB dataset from whole-brain larval zebrafish recordings [Ahrens 2013], his algorithms run in the order of minutes using 30 machines. He proposed using legacy software systems such as MATLAB to convert any legacy data to a simple common data format (rows of time series) before feeding into Thunder.

• J. Freeman, N. Vladimirov, T. Kawashima, Y. Mu, D. V. Bennett, J. Rosen, C. Yang, L. Looger, M.Ahrens. Mapping the brain at scale. COSYNE 2014

• Ahrens, M. B., Orger, M. B., Robson, D. N., Li, J. M., and Keller, P. J. (2013). Whole-brain functional imaging at cellular resolution using light-sheet microscopy. Nature Methods, 10(5):413-420.

### Mark Churchland – Tensor-based dimensionality reduction for data exploration

What drives the temporal structure of a neural population response $x(t)$: stimulus or (linear) dynamics? Mark proposed a method to answer this question by decomposing a time-by-neuron-by-condition data array with linear combination of small number of either R-sheets (condition-by-time) or D-sheets (neuron-by-time). If $x(t)$ is linearly driven by  stimulus through tuning curve $x(t) = Bu(t)$, a fixed number R-sheets will approximate the neural response well, no matter how long the recordings are. On the other hand, if dynamics drive the neural activity well, e.g., $x(t) = Ax(t-1)$, then a fixed number of D-sheets will continue to explain the data well. Hence, he argued, the decay in low-rank approximation performance as more time is included gives insight into the fundamental driver of neural response: dynamics or stimulus. He showed evidence that in the motor cortex, dynamics alone explains the data better.

• Jeffrey Seely, Matthew T Kaufman, John Cunningham, Stephen Ryu, Krishna Shenoy, Mark Churchland. Dimensionality in motor cortex: differences between models and experiment. COSYNE 2012

• Jeffrey Seely, Matthew Kaufman, Adam Kohn, Matthew Smith, Anthony Movshon, Nicholas Priebe, Stephen Lisberger, Stephen Ryu, Krishna Shenoy, Larry Abbott. Quantifying representational and dynamical structure in large neural datasets. COSYNE 2013

### John Cunningham – Generic linear dimensionality reduction for high-dimensional neural data

John began by listing 3 advantages of high-dimensional neural data: (1) single trial statistical power (average over population instead of trials), (2) population response structure (population code and dynamics), (3) exploratory data analysis (generate systems level hypotheses). He then turned to consider a particular technique: linear dimensionality reduction.

Linear dimensionality reduction is a common tool in statistics, data analysis, and many other fields. This term includes tools such as PCA, LDA and CCA. Standard practice in the use of these techniques is to perform an eigendecomposition of some matrix, and then truncate some of its eigenvalues. However, John points out that this common practice often does not maximize the objective function associated with a given technique. Only in the case of PCA is it provably optimal for minimizing $|| X - BB^\top X ||^2$. John proposed using manifold optimization techniques instead. He demonstrated several cases where optimization outperforms eigendecomposition truncation for linear dimensionality reduction.

• Absil, Pierre-Antoine, Robert Mahony, and Rodolphe Sepulchre. “Optimization on manifolds.” (2007).

### Rob Kass – Statistical considerations in making inferences about neural networks: The case of synchrony detection

Rob Kass dealt with the problem of identifying excess synchrony, or discovering connectivity structure from data [Kelly & Kass 2012]. He took advantage of the fact that the number of interactions increases quadratically as the number of neurons increased–he could reliably estimate the alternate hypothesis of the connection strength (or synchrony) using Bayesian control of false discovery. Basically, the estimated distribution of connectivity resulting from point process regression was modeled as a mixture of theoretical null hypothesis and samples from the alternative hypothesis. He also included covariates such as population level oscillation (via population rate) in his model.

• Kelly, R. C. and Kass, R. E. (2012). A framework for evaluating pairwise and multiway synchrony among Stimulus-Driven neurons. Neural Computation, 24(8):2007-2032.

• Emery Brown, Uri Eden, Rob Kass. (2014) Analysis of Neural Data. [amazon]

### Lars Büsing – Dynamical component analysis of neural data

Latent-variable dynamical models are an increasingly popular tool for large-scale neural population analysis. A popular technique under this framework is to assume that spikes are Poisson-distributed with rates given by some function of a linearly-evolving latent state. While this technique is closely related to traditional Kalman filtering algorithms, the Poisson likelihood renders them difficult to compute. Lars presented a new technique for performing approximate inference under the Poisson observation model, as well a new technique for clustering neural populations with the same dynamics.

• Lars Buesing, Maneesh Sahani, Jakob H. Macke. Spectral learning of linear dynamics from generalised-linear observations with application to neural population data. NIPS 2012

• Khan, Aravkin, Friedlander, Seeger, “Fast Dual Variational Inference for Non-Conjugate Latent Gaussian Models.” Proceedings of the 30th International Conference on Machine Learning (ICML). No. EPFL-CONF-186800. Omni Press, 2013.

• Lars Buesing. Unsupervised identification of excitatory and inhibitory populations from multi-cell recordings. COSYNE 2014

### Matteo Carandini – Soloists and choristers in a cortical population

Matteo started by emphasizing that many widely-used models of neural data do not scale well with the number of parameters, and often do not have a clear biological interpretation. He then considered the raster marginal model of [Okun et al. 2012], a class models designed to be consistent with two measurements: (1) the marginal firing rate for each neuron, and (2) the population rate distribution (the number of neurons firing in each time bin). He showed that while many neurons in the population are choristers (are highly correlated with the population rate), a few are soloists. Soloists and choristers are indistinguishable, except that choristers are more “enthusiastic” (respond with an elevated firing rate in response to stimuli). The raster marginal model cannot distinguish between soloists and choristers, and so Matteo proposed an extension that includes a new statistic: (3) the correlation between each neuron and the population rate. His final model, including statistics (1), (2), and (3), has at most 3N parameters (where N is the number of neurons). This model captures the distinction between soloists and choristers, as well as correlation coefficients and connection probability. Matteo suggested using this model as a null hypothesis. If the model explains a potentially interesting phenomenon — perhaps it is not an interesting phenomenon after all.

• Okun, M., Yger, P., Marguet, S. L., Gerard-Mercier, F., Benucci, A., Katzner, S., Busse, L., Carandini, M., and Harris, K. D. (2012). Population rate dynamics and multineuron firing patterns in sensory cortex. The Journal of neuroscience, 32(48):17108–17119.

• Thomas Mrsic-Flogel. The functional organisation of synaptic connectivity in visual cortical microcircuits. COSYNE 2014

### I. Memming Park – Scaling up with Bayes

(Unfortunately, Matthias Bethge couldn’t make it, so Memming gave a talk instead.)
To make inferences with high-dimensional data, we need strong assumptions. In Bayesian statistics, we encode our assumptions and prior knowledges into the prior distribution. In this talk, Memming showed how to use popular parametric models of spike word distributions, and use Bayesian nonparametrics techniques to make them flexible.

• Il Memming Park, Evan Archer, Kenneth Latimer, Jonathan Pillow. Universal models for binary spike patterns using centered Dirichlet processes. NIPS 2013

• Ganmor, E., Segev, R., and Schneidman, E. (2011). Sparse low-order interaction network underlies a highly correlated and learnable neural population code. Proceedings of the National Academy of Sciences, 108(23):9679-9684.

### Surya Ganguli – What are the right questions, and how many neurons do we need to answer them?

Surya’s talk introduced a new quantity, called “task complexity”, which bounds the dimensionality of neural population dynamics. This talk built on the talk given by Perian Gao’s given in the main meeting. Task complexity is an upper bound of trial-averaged neural response, characterized by the degree of freedom in the task conditions and correlation time scale of neural responses (there were hot discussions concerning whether  if “task complexity” is a misleading notion, since it includes the smoothness of the neural trajectory for the task). Surya claimed that the task complexities of many of the experiments are such that it is not surprising to recover low-dimensional dynamics from observations from 100 neurons in motor cortex. He emphasized that we need to design experiments with higher task complexity to take advantage of high-dimensional neural data. Surya also suggested that we may not need to observe many neurons to recover the dynamics, under certain conditions provided by the theory of random projections.

• Peiran Gao, E. Trautmann, B. Yu, G. Santhanam, S. Ryu, K. Shenoy, S. Ganguli. A theory of neural dimensionality and measurement. COSYNE 2014

• Surya Ganguli. A theory of neural dimensionality, dynamics and measurement: the neuroscientist and the single neuron. NIPS workshop 2013

## 5 thoughts on “Scalable models workshop recap”

1. Reblogged this on Memming and commented:

Evan and I wrote a summary of the COSYNE 2014 workshop we organized!

2. xcorr says:

Reblogged this on xcorr: comp neuro and commented:
Here’s a recap from the Scaling models for high-dimensional neural data at this year’s Cosyne. Unfortunately I couldn’t make it – I’m defending in a few days – but there were some really interesting talks.

I’m especially intrigued by Jeremy Freeman’s analysis of zebrafish whole-brain recordings. He has a tentalizing talk at a tech conference on this subject:

His analysis toolbox is available here.

and some stuff from


text:
# Conditions in which the sign of a coefficient will change between a linear probability model and a logistic model

I am estimating a model where the DV is a binary variable and the key independent variable is the interaction between a dummy variable and a continuous variable. I am getting an very odd result where the sign of the coefficient switches between a positive and negative value when I go from a LPM to a Logit model, (though both remain insignificant.)

I've searched all over my data for a coding error, (which my adviser said is the most likely culprit), but I can't find anything amiss. Are there any other conditions that might explain this? I am using robust standard errors in both models.

Thank you for your help! Also apologies if I'm not including enough information, this is my first time with this site.

UPDATE:

As per @Isabella-Ghement's comment in the answer, here are the plots I produced for both the LPM and Logit models she described in the answer.

Actual Data, (black), vs. LPM, (green) vs. Logit, (red) , Dummy=0:

Actual Data, (black), vs. LPM, (green) vs. Logit, (red) , Dummy=1:

Update 2:

Here is the qqplot for the LPM:

And for the logit model, (from DHARMa):

• They’re two different estimation methods that will generally yield different results. So it’s not that surprising that they could yield different signs for an irrelevant predictor, for example. – The Laconic Mar 22 at 2:04

There's nothing like plotting your data to get some insights into what might be going on.

For the LPM model (linear probability model), you want to plot your DV versus CONT separately for each of the values of DUMMY, where CONT is the continuous predictor and DUMMY is the dummy predictor variable. When DUMMY = 0, the plot of DV against CONT would look similar to the first plot listed on this page: How to plot binary (presence/absence - 1/0) data against continuous variables. When DUMMY = 1, the plot will change appearance but it will follow the same visual style. For both plots, DV should be plotted on the Y axis and CONT on the X axis.

The LPM model can be stated like:

DV = beta0 + beta1*CONT + beta2*DUMMY + beta3*CONT*DUMMY + epsilon

After fitting this model to the data, you can obtain the estimated values of the coefficients beta0, beta1, beta2 and beta3 - let's call them b0 through b3. Using this information, you can add a fitted regression line to each of your two plots. That line will depict how the estimated probability that DV = 1 changes with the values of CONT when DUMMY = 0 and when DUMMY = 1, respectively. The end plots should look similar to the plot located in the left panel of the first figure found in the document available at https://are.berkeley.edu/courses/EEP118/fall2010/section/13/Section%2013%20Handout%20Solved.pdf (except the plots will show both the data and the fitted probability line).

The fitted probability lines produced by the LPM model can be obtained by plotting in the corresponding plot of DV versus CONT:

1. b0 + b1*CONT versus CONT for DUMMY = 0;

2. (b0 + b2) + (b1 + b3)*CONT versus CONT for DUMMY = 1.

Since your LPM model includes an interaction term, you would expect the two lines to potentially have different slopes.

Now, each of these two plots can be enhanced by adding the corresponding nonlinear fitted probability curve produced by the binary logistic regression model. This model can be stated as:

log (Odds that DV is equal to 1) = gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY

and then re-expressed as:

Prob that DV is equal to 1 = exp(gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY)/
[1 + exp(gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY)]

If you fit the binary logistic model (as expressed in its log odds formulation) to the data and get the estimated values of its coefficients - let's call them g0 through g3 - then you can plot the following fitted probability curves in your plots:

The fitted probability lines produced by the LPM model can be obtained by plotting the following:

1. exp(g0 + g1*CONT)/[1 + exp(g0 + g1*CONT)] versus CONT for DUMMY = 0;

2. exp((g0 + g2) + (g1 + g3)*CONT)/[1 + exp((g0 + g2) + (g1 + g3)*CONT)] versus CONT for DUMMY = 1;

in the appropriate plot of DV versus CONT.

The fitted probability curves should look similar to the plot located in the right panel of the first figure found in the document available at https://are.berkeley.edu/courses/EEP118/fall2010/section/13/Section%2013%20Handout%20Solved.pdf (except the plots will also show both the data and the fitted probability line produced by the LPM model).

You should be on the lookout for two types of patterns in your data for each plot (recall that one plot corresponds to DUMMY = 0 and the other to DUMMY = 1).

A. Most of the 0 values for the DV variable are clumped at the lower end of the range of CONT values, and most of the 1 values for the DV variable are clumped at the higher end of the range of CONT values:

DV = 1:          ooooooooooooo

DV = 0:   ooooooooooo

---- CONT ---------->

B. Most of the 0 values for the DV variable are clumped at the higher end of the range of CONT values, and most of the 1 values for the DV variable are clumped at the lower end of the range of CONT values:

DV = 1:   ooooooooooooo

DV = 0:             ooooooooooo

---- CONT ---------->

In scenario A, you would expect CONT to have a positive linear effect on the probability that Y = 1 for the LPM model and a positive nonlinear effect on the same probability.

In scenario B, you would expect CONT to have a negative linear effect on the probability that Y = 1 for the LPM model and a negative nonlinear effect on the same probability.

In general, I agree with your supervisor that if you encounter patterns such as the ones described in the above scenarios in your plots, you would intuitively expect the fitted probability line produced by LPM and the fitted probability curve produced by the logistic regression model to follow the same direction (e.g., both positive or both negative). If that is not the case, one of the models is not appropriate for the data - most likely, the LPM model.

In any event, plotting your data will reveal if you have any other issues in the DV data that might affect the LPM model fit - for instance, the majority of your DV data values are equal to 0 (or perhaps the majority are equal to 1). Or there could be issues with your CONT variable - too narrow a range of values; only a few distinct values; outliers or gaps in its distribution; etc.

• This is so helpful! Thank you Isabella! – StanO Mar 25 at 2:35
• Just a small clarification, did you mean to say that in plot 2 for both the LPM and the Logit, that DUMMY=1, not DUMMY=0? That appears to be the case, but I just wanted to double check. – StanO Mar 26 at 2:20
• Good catch! That's exactly what I meant and I edited my answer to reflect that plot 2 for both models should use DUMMY = 1. – Isabella Ghement Mar 26 at 3:02
• You can post your plots in your question as an addendum if you want - then we can see better what is going on. – Isabella Ghement Mar 26 at 3:09
• Yeah, when I strip the other controls from the model, the coefficient signs for both models are the same and, as the above plots show, the fits agree. However, when I add other controls, the signs flip and the problems I described above begin. I'm sorry, I should have been more clear on that. – StanO Mar 27 at 0:25


text:
Skip to main content

# Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience

Comşa, Ioan-Sorin, Saleme, Estêvão Bissoli, Covaci, Alexandra, Assres, Gebremariam Mesfin, Trestian, Ramona, Saibel Santos, Celso Alberto, Ghinea, Gheorghiţă (2019) Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience. IEEE MultiMedia, 27 (1). pp. 27-36. ISSN 1070-986X. (doi:10.1109/MMUL.2019.2954405) (KAR id:79189)

 PDF Publisher pdf Language: English Download (743kB) Preview Preview This file may not be suitable for users of assistive technology. Request an accessible format Official URLhttps://doi.org/10.1109/MMUL.2019.2954405%E2%80%8B

## Abstract

One of the main challenges in current multimedia networking environments is to find solutions to help accommodate the next generation of mobile application classes with stringent Quality of Service (QoS) requirements whilst enabling Quality of Experience (QoE) provisioning for users. One such application class, featured in this paper, is $$360^∘$$ mulsemedia-multiple sensorial media-which enriches $$360^∘$$ video by adding sensory effects that stimulate human senses beyond those of sight and hearing, such as the tactile and olfactory ones. In this paper, we present a conceptual framework for $$360^∘$$ mulsemedia delivery and a $$360^∘$$ mulsemedia-based prototype that enables users to experience $$360^∘$$ mulsemedia content. User evaluations revealed that higher video resolutions do not necessarily lead to the highest QoE levels in our experimental setup. Therefore, bandwidth savings can be leveraged with no detrimental impact on QoE.

Item Type: Article 10.1109/MMUL.2019.2954405 Streaming media, Quality of experience, Bandwidth, Prototypes, Synchronization, Video recording T Technology > TK Electrical engineering. Electronics. Nuclear engineering Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Engineering and Digital Arts Alexandra Covaci 06 Dec 2019 14:26 UTC 16 Feb 2021 14:10 UTC https://kar.kent.ac.uk/id/eprint/79189 (The current URI for this page, for reference purposes) https://orcid.org/0000-0002-3205-2273
• Depositors only (login required):

### Downloads

Downloads per month over past year


text:
Mironenko reflecting function

The reflecting function ${\displaystyle \,F(t,x)}$ of a dynamical system connects the past state ${\displaystyle \,x(-t)}$ of it with the future state ${\displaystyle \,x(t)}$ of it by the formula ${\displaystyle \,x(-t)=F(t,x(t)).}$ The concept of the reflecting function was introduсed by Uladzimir Ivanavich Mironenka.

Definition

For the differential system ${\displaystyle {\dot {x}}=X(t,x)}$ with the general solution ${\displaystyle \varphi (t;t_{0},x)}$ in Cauchy form Reflecting Function is defined by formula ${\displaystyle F(t,x)=\varphi (-t;t,x).}$

Application

If a vector-function ${\displaystyle X(t,x)}$ is ${\displaystyle \,2\omega }$-periodic with respect to ${\displaystyle \,t}$, then ${\displaystyle \,F(-\omega ,x)}$ is the in-period ${\displaystyle \,[-\omega ;\omega ]}$ transformation (Poincaré map) of the differential system ${\displaystyle {\dot {x}}=X(t,x).}$ Therefore the knowledge of the Reflecting Function give us the opportunity to find out the initial dates ${\displaystyle \,(\omega ,x_{0})}$ of periodic solutions of the differential system ${\displaystyle {\dot {x}}=X(t,x)}$ and investigate the stability of those solutions.

For the Reflecting Function ${\displaystyle \,F(t,x)}$ of the system ${\displaystyle {\dot {x}}=X(t,x)}$ the basic relation

${\displaystyle \,F_{t}+F_{x}X+X(-t,F)=0,\qquad F(0,x)=x.}$

is holding.

Therefore we have an opportunity sometimes to find Poincaré map of the non-integrable in quadrature systems even in elementary functions.


text:
# Rowspace and kernel

by UrbanXrisis
Tags: kernel, rowspace
 P: 1,200 $$A = \left(\begin{array}{cccc}-1 &6&5&9 \\ -1&0&1&3 \end{array}\right)$$ Find orthonormal bases of the kernel, row space. To find the bases, I did reduced the array to its RREF. $$A = \left(\begin{array}{cccc}1 & 0&-1&-3\\ 0&1&2/3&1 \end{array}\right)$$ Then the orthonormal bases would just be that divided by the length. $$||v_1||=\sqrt{1+1+3^2}=\sqrt{11}$$ $$||v_2||=\sqrt{1+(2/3)^2+1}=\sqrt{2.44444}$$ so that means, the orthonormal bases would be: $$A = \left(\begin{array}{cccc} \frac{1}{ \sqrt{11}} & 0&\frac{-1}{ \sqrt{11}}&\frac{-3}{ \sqrt{11}} \\0 & \frac{1}{ \sqrt{2.44444}} & \frac{.66666}{ \sqrt{2.44444}} &\frac{1}{ \sqrt{2.44444}}\end{array}\right)$$ what exactly is the orthonormal bases of the kernel? Also, isnt the row space the same as the vectors of the bases? I think I also did something wrong in my calculations
 P: 1,200 is what i did above the orthonormal row space? that is wrong as well, i dont know why.... however: using the Gram-Schmidt process, i still get an error: $$A = \left(\begin{array}{cccc}-1 &6&5&9 \\ -1&0&1&3 \end{array}\right)=\left(\begin{array}{cc}W_1 &W_2 \end{array}\right)$$ want to find an orthonormal basis $$R={U_1 ,U_2}$$ $$U_1=\frac{W}{||W_1||}$$ $$||W_1||=\sqrt{11}$$ $$U_1=\frac {\left(\begin{array}{cccc}-1 &6&5&9 \end{array}\right)}{\sqrt{11}}$$ $$U_2=\frac{W_2- U_1}{||W_2- U_1||}$$ where $$W_2=\left(\begin{array}{cccc} -1&0&1&3 \end{array}\right)$$ $$U_1=\frac {\left(\begin{array}{cccc}-1 &6&5&9 \end{array}\right)}{\sqrt{11}}$$ $$U_1=\left(\begin{array}{cccc}-1/\sqrt{11} &6\sqrt{11}&5\sqrt{11}&9\sqrt{11} \end{array}\right)$$ is the the correct set up to get an orthonormal basis? Also, to get an orthonormal row space, what would I have to do?


text:
## controlled thermodynamic integral for Bayesian model comparison [reply]

Posted in Books, pictures, Running, Statistics, University life with tags , , , , , , , , , , , , on April 30, 2014 by xi'an

Chris Oates wrotes the following reply to my Icelandic comments on his paper with Theodore Papamarkou, and Mark Girolami, reply that is detailed enough to deserve a post on its own:

Thank you Christian for your discussion of our work on the Og, and also for your helpful thoughts in the early days of this project! It might be interesting to speculate on some aspects of this procedure:

(i) Quadrature error is present in all estimates of evidence that are based on thermodynamic integration. It remains unknown how to exactly compute the optimal (variance minimising) temperature ladder “on-the-fly”; indeed this may be impossible, since the optimum is defined via a boundary value problem rather than an initial value problem. Other proposals for approximating this optimum are compatible with control variates (e.g. Grosse et al, NIPS 2013, Friel and Wyse, 2014). In empirical experiments we have found that the second order quadrature rule proposed by Friel and Wyse 2014 leads to substantially reduced bias, regardless of the specific choice of ladder.

(ii) Our experiments considered first and second degree polynomials as ZV control variates. In fact, intuition specifically motivates the use of second degree polynomials: Let us presume a linear expansion of the log-likelihood in θ. Then the implied score function is constant, not depending on θ. The quadratic ZV control variates are, in effect, obtained by multiplying the score function by θ. Thus control variates can be chosen to perfectly correlate with the log-likelihood, leading to zero-variance estimators. Of course, there is an empirical question of whether higher-order polynomials are useful when this Taylor approximation is inappropriate, but they would require the estimation of many more coefficients and in practice may be less stable.

(iii) We require that the control variates are stored along the chain and that their sample covariance is computed after the MCMC has terminated. For the specific examples in the paper such additional computation is a negligible fraction of the total computational, so that we did not provide specific timings. When non-diffegeometric MCMC is used to obtain samples, or when the score is unavailable in closed-form and must be estimated, the computational cost of the procedure would necessarily increase.

For the wide class of statistical models with tractable likelihoods, employed in almost all areas of statistical application, the CTI we propose should provide state-of-the-art estimation performance with negligible increase in computational costs.

## controlled thermodynamic integral for Bayesian model comparison

Posted in Books, pictures, Running, Statistics, University life with tags , , , , , , , , , , on April 24, 2014 by xi'an

Chris Oates, Theodore Papamarkou, and Mark Girolami (all from the University of Warwick) just arXived a paper on a new form of thermodynamic integration for computing marginal likelihoods. (I had actually discussed this paper with the authors on a few occasions when visiting Warwick.) The other name of thermodynamic integration is path sampling (Gelman and Meng, 1998). In the current paper, the path goes from the prior to the posterior by a sequence of intermediary distributions using a power of the likelihood. While the path sampling technique is quite efficient a method, the authors propose to improve it through the recourse to control variates, in order to decrease the variance. The control variate is taken from Mira et al. (2013), namely a one-dimensional temperature-dependent transform of the score function. (Strictly speaking, this is an asymptotic control variate in that the mean is only asymptotically zero.) This control variate is then incorporated within the expectation inside the path sampling integral. Its arbitrary elements are then calibrated against the variance of the path sampling integral. Except for the temperature ladder where the authors use a standard geometric rate, as the approach does not account for Monte Carlo and quadrature errors. (The degree of the polynomials used in the control variates is also arbitrarily set.) Interestingly, the paper mixes a lot of recent advances, from the zero variance notion of Mira et al. (2013) to the manifold Metropolis-adjusted Langevin algorithm of Girolami and Calderhead (2011), uses as a base method pMCMC (Jasra et al., 2007). The examples processed in the paper are regression (where the controlled version truly has a zero variance!) and logistic regression (with the benchmarked Pima Indian dataset), with a counter-example of a PDE interestingly proposed in the discussion section. I quite agree with the authors that the method is difficult to envision in complex enough models. I also did not see mentions therein of the extra time involved in using this control variate idea.

## Vanilla Rao-Blackwellisation [re]revised

Posted in R, Statistics with tags , , , , , , , on June 1, 2010 by xi'an

Although the revision is quite minor, it took us two months to complete from the time I received the news in the Atlanta airport lounge… The vanilla Rao-Blackwellisation paper with Randal Douc has thus been resubmitted to the Annals of Statistics. And rearXived. The only significant change is the inclusion of two tables detailing computing time, like the one below

$\left| \begin{matrix} \tau &\text{median} &\text{mean }&q_{.8} &q_{.9} &\text{time}\\ 0.25 &0.0 &8.85 &4.9 &13 &4.2\\ 0.50 &0.0 &6.76 &4 &11 &2.25\\ 1.00 &0.25 &6.15 &4 &10 &2.5\\ 2.00 &0.20 &5.90 &3.5 &8.5 &4.5\\\end{matrix} \right|$

which provides different evaluations of the additional computing effort due to the use of the Rao–Blackwellisation: median and mean numbers of additional iterations, $80\%$ and $90\%$ quantiles for the additional iterations, and ratio of the average R computing times obtained over $10^5$ simulations. (Turning the above table into a formula acceptable by WordPress took me for ever, as any additional white space between the terms of the matrix is mis-interpreted!) Now, the mean time column does not look very supportive of the Rao-Blackwellisation technique, but this is due to the presence of a few outlying runs that required many iterations before hitting an acceptance probability of one. Excessive computing time can be curbed by using a pre-set number of iterations, as described in the paper…

## pimax(mcsm)

Posted in Books, R, Statistics with tags , , , , on May 13, 2010 by xi'an

The function pimax from our package mcsm is used in to reproduce Figure 5.11 of our book Introducing Monte Carlo Methods with R. (The name comes from using the Pima Indian R benchmark as the reference dataset.) I got this email from Josué

I ran the ‘pimax’ example from the mcsm manual, and it gave me the following message:

> pimax(Nsim = 10^3)
Error in raaj[t, ] = apply(as.matrix(aas), 1, margap) :
number of items to replace is not a multiple of replacement length
> pimax()
Error in raaj[t, ] = apply(as.matrix(aas), 1, margap) :
number of items to replace is not a multiple of replacement length

but when running pimax(10^2) on my machine I did get the following picture and no error message. So I wonder if this is a matter of version of R or something else…

## Vanilla Rao-Blackwellisation for revision

Posted in R, Statistics with tags , , , , , on March 18, 2010 by xi'an

The vanilla Rao-Blackwellisation paper with Randal Douc that had been resubmitted to the Annals of Statistics is now back for a revision, with quite encouraging comments:

The paper has been reviewed by two referees both of whom comment on the clear exposition and the novelty of the results. Both referees point to the empirical results as being suggestive of a more incremental improvement in practice rather than a major advance. However the approach the authors adopt is novel and I believe may motivate further developments in this area.

I cannot but agree on those comments! Since we are reducing the variance of the weights, the overall effect may be difficult to spot in practical applications. In the current version of the paper, we manage 20% reduction in the variance of those weights, but obviously this does not transfer to the same reduction of the variance of the overall estimator! Our vanilla Rao-Blackwellisation does not speed up the Markov chain.


text:
2014
01-26

# Fire-Control System

A new mighty weapon has just been developed, which is so powerful that it can attack a sector of indefinite size, as long as the center of the circle containing the sector is the location of the weapon. We are interested in developing a fire-control system that calculates firing-solutions automatically.
The following example gives an example of a firing solution:
Figure 1

Here the firing region is the sector "ABC" that covers six points: A, B, C, D, E, H. You may further assume that the weapon is always located at point (0, 0), no targets will be on the point (0, 0) and the coordinates of the targets will be distinct.
A firing solution is called effective if and only if it covers a minimum of K points
out of N given points (targets) on the two-dimensional Cartesian plane. Furthermore,since the cost of a particular fire solution is in direct proportion to the size of the area it covers, a firing could be quite costly; thus we are only interested in the optimal firing solution with the minimum cost.

There are multiple test cases in the input file.
Each test case starts with two non-negative integers, N and K
(1 ≤ N ≤ 5000 , K ≤ N ), followed by N lines each containing two integers, X, and Y, describing the distinct location of one target. It is guaranteed that the absolute value of any integer does not exceed 1000.
Two successive test cases are separated by a blank line. A case with N = 0 and K = 0 indicates the end of the input file, and should not be processed by your program.

There are multiple test cases in the input file.
Each test case starts with two non-negative integers, N and K
(1 ≤ N ≤ 5000 , K ≤ N ), followed by N lines each containing two integers, X, and Y, describing the distinct location of one target. It is guaranteed that the absolute value of any integer does not exceed 1000.
Two successive test cases are separated by a blank line. A case with N = 0 and K = 0 indicates the end of the input file, and should not be processed by your program.

3 1
0 1
1 0
-5 -6
3 2
0 2
2 0
-5 -6
0 0

Case #1: 0.00
Case #2: 3.14

#include <iostream>
#include <stdio.h>
#include <queue>
#include <algorithm>
#include <math.h>
#include <string.h>
using namespace std;
#define N 100100
#define PI (2*asin(1.0))

struct node
{
int x,y;
double r;//表示半径
double du;//用来表示角度
}g[N];

int n,k;
double g1[N];

int cmp(node t,node t1)
{
return t.du<t1.du;
}

double que[2*N];

double mabs(double x)
{
if(x<0) return -x;
return x;
}

int main()
{
//freopen("//home//ismdeep//xianchang1//in","r",stdin);
int tt=1;
double pi=PI;
while(scanf("%d%d",&n,&k)&&(n+k))
{

for(int i=1;i<=n;i++)
{
scanf("%d%d",&g[i].x,&g[i].y);
g[i].r=sqrt((double)g[i].x*g[i].x+g[i].y*g[i].y);
g1[i]=g[i].r;

double tmp;

if(mabs(g[i].x-0)<1e-8)
{
if(g[i].y>0) tmp=90.0;
else tmp=270.0;
}
else
{
tmp=((double)g[i].y/(double)g[i].x); //将这个点的斜率求出来
if(mabs(g[i].y-0)<1e-8)
{
if(g[i].x>0) tmp=0.0;
else tmp=180.0;
}
else
{
tmp=atan(tmp);//求出角度
tmp=(180.0/PI)*tmp;
if(g[i].y*g[i].x > 0&&g[i].y<0)
tmp+=180.0;
if(g[i].y*g[i].x<0)
{
tmp*=-1;
if(g[i].x<0)
tmp=180.0-tmp;
else
{
tmp=360-tmp;
}
}

}
}
g[i].du=tmp;
}
// 角度求好了
if(k==0)
{
printf("Case #%d: 0.00\n",tt++);
continue;
}
//sort(g1+1,g1+n+1); //半径从小到大来搞一搞啊
sort(g+1,g+1+n,cmp);

double mi=1999999999;

for(int ii=1;ii<=n;ii++)
{
double key=g1[ii]; //表示固定的半径
int flag=0;

for(int i=1;i<=n;i++)//提前
{
if(g[i].r <= key+1e-6)
{
que[flag++]=g[i].du;
}
}

for(int i=0;i<flag;i++)
{
que[flag+i] = que[i]+360.0;
}

if(flag<k) continue;

for(int i=0;i<flag;i++)
{
double tmp=que[i];
int tt=i+k-1;
double tmp1=que[tt];
tmp=tmp1-tmp;

tmp=(tmp/(360.0))*PI*key*key;

mi=min(tmp,mi);
}
}

printf("Case #%d: ",tt++);

printf("%.2lf\n",mi);
}
return 0;
}

1. 5.1处，反了；“上一个操作符的优先级比操作符ch的优先级大，或栈是空的就入栈。”如代码所述，应为“上一个操作符的优先级比操作符ch的优先级小，或栈是空的就入栈。”

2. 这道题目虽然简单，但是小编做的很到位，应该会给很多人启发吧！对于面试当中不给开辟额外空间的问题不是绝对的，实际上至少是允许少数变量存在的。之前遇到相似的问题也是恍然大悟，今天看到小编这篇文章相见恨晚。