PhD C: What channel am I onPhD E: Channel.PhD C: Oh, channel two.PhD G: Make sure to turn your microphone on.PhD E: Channel.PhD G: There's a battery.Grad B: There we go.PhD G: OK. Your channel number's already on this blank sheet.Grad B: Yeah.PhD G: So you just {disfmarker} If you can {disfmarker}Professor F: Channel five Channel five.PhD E: Channel whatever.Professor F: I'm on channel five.Grad B: Camera one, camera two.PhD E: What am IProfessor F: Little lowPhD E: Channel fourProfessor F: Channel five.PhD E: This number four OK.Professor F: Channel five. OK.PhD G: The gai the gain's up at it {disfmarker} what it usually is,Professor F: Is itPhD G: but if you think it's {disfmarker} Yeah. It's sort of a default. But I can set it higher if you like.Professor F: Oh. Maybe it should be a little higher.PhD G: YeahProfessor F: It's not showing much. Test, test, test, test, test, test, test, test, test, test. OK, that {disfmarker} that seems better Yeah OK, good. Ah, that's good, that's good. That's Ahh. Mmm. So I {disfmarker} I had a question for Adam. Have we started alreadyPhD G: Well, we started recording, but {disfmarker} Yeah.Professor F: Yeah. Is Jane around or {disfmarker}PhD D: I saw her earlier.Professor F: Uh.PhD D: I think {disfmarker}PhD G: She can just walk in, I guess, or {disfmarker}PhD D: Yeah. She'll probably come up.Professor F: Right.PhD G: Since we're starting late I figured we'd better just start.Professor F: Yeah. Great idea. I was gonna ask Adam to, uh, say if he thought anymore about the demo stuff because {vocalsound} it occurred to me that this is late May and the DARPA meeting is in {pause} mid July. Uh, but {vocalsound} I don't remember w what we {disfmarker} I know that we were gonna do something with the transcriber interface is one thing, but I thought there was a second thing. Anybody rememberPhD G: Well, we were gonna do a mock - up, like, question answering or something, I thought, that was totally separate from the interface. Do you remember Remember, like, asking questions and retrieving, {vocalsound} but in a pre - stored fashion.Professor F: Mm - hmm. Right.PhD G: That was the thing we talked about, I think, before the transcriber {disfmarker}Professor F: Yeah.PhD G: Come on in.Professor F: Alright. So anyway, you have to sort out that out and get somebody going on it cuz we're {disfmarker} got a {disfmarker} got a month left basically. So.PhD G: You like these. Right OK, good.Professor F: OK. Um OK. So, what are we g else we got You got {disfmarker} you just wrote a bunch of stuff.PhD G: No. That was all, um, previously here.Professor F: Oh.PhD G: I was writing {pause} the digits and then I realized I could xerox them,Professor F: Oh, oh.PhD G: because I didn't want people to turn their heads from these microphones. So. We all, by the way, have the same digit form, for the record. So.Professor F: That's cool.PhD G: Yeah.Professor F: So, the choice is, uh, which {disfmarker} which do we want more, the {disfmarker} the {disfmarker} the comparison, uh, {vocalsound} of everybody saying them at the same time or the comparison of people saying the same digits at different times that {disfmarker}PhD G: It's just cuz I didn't have any more digit sheets.Professor F: I know that. But, you know, which opportunity should wePhD G: So. Yeah.PhD C: Unison.Professor F: exploit {vocalsound} Unison.PhD G: I mean, it {disfmarker} Actually it might be good to have them separately and have the same exact strings. I mean, we could use them for normalizing or something, but it of course goes more quickly doing them in unison.Professor F: I guess we'll seePhD G: I don't know.Professor F: i I guess it's dependent onPhD G: See how long we go.Professor F: how long we go and how good the snack is out there.PhD E: But anyway, they won't be identical as somebody is saying zero in some {disfmarker} sometimes, you know, saying O, and so, it's not i not identical.Professor F: Yeah. Hmm. Get some advance intelligence.PhD G: Right. Right.Professor F: Yeah. We'd have to train.PhD G: We'd be like a chorus.PhD E: OK.Professor F: Yeah. We'd have to get s get some experience.PhD C: Greek chorus.Grad B: Yeah.PhD G: Yes.Professor F: Yeah. Really {pause} boring chorus. Um. Do we {pause} have an agenda Adam usually tries to put those together, but he's ill.PhD D: I've got a couple of things to talk about.Professor F: So. Yeah. Uh ju what {disfmarker} what might those bePhD D: Uh, IBM stuff and, um, just getting {vocalsound} uh, meeting information organized.Professor F: Meeting info organized. OK. Um.PhD C: Are you implying that it's currently disorganizedPhD D: In my mind.Professor F: Is there stuff that's happened about, um, uh, the {pause} SRI recognizer et cetera, tho those things that were happening before with {disfmarker}PhD C: Well.Professor F: Y y you guys were doing a bunch of experiments with different front - ends and then with {disfmarker} Is {disfmarker} is that still sort of where it was, uh, the other dayPhD C: We're improving.Professor F: We're improving.PhD C: Yeah.PhD D: Now the {disfmarker} the {disfmarker} You saw the note that the PLP now is getting basically the same as the MFCC.Professor F: Right.PhD D: RightPhD C: Yeah. Actually it looks like it's getting better.Professor F: Right. Oh.PhD C: So. But {disfmarker} but it's not {disfmarker}Professor F: Just with {disfmarker} with age, kind of.PhD C: With age. Yeah.Professor F: Yeah. Yeah.PhD C: But, uh, that's not d directly related to me. Doesn't mean we can't talk about it. Um, it seems {disfmarker} It looks l I haven't {disfmarker} The {disfmarker} It's {disfmarker} The experiment is still not complete, but, um, it looks like the vocal tract length normalization is working beautifully, actually, w using the warp factors that we computed for the SRI system and just applying them to the {vocalsound} ICSI front - end.Professor F: Mm - hmm. That's pretty funny.PhD C: Yeah.Professor F: OK.PhD G: So you just need to {pause} copy over to this one.PhD C: Just had to take the reciprocal of the number because they have different meanings in the two systems.Postdoc A: OK.Professor F: Ah! Yeah. Well, that's always good to do.PhD C: Yeah.Professor F: OK. OK. Uh {disfmarker}PhD C: But one issue actually that just came up in discussion with Liz and {disfmarker} and Don was, um, as far as meeting recognition is concerned, um, we would really like to, uh, move, uh, to, uh, doing the recognition on automatic segmentations.Professor F: Yeah.PhD C: Because in all our previous experiments, we had the {disfmarker} uh, you know, we were essentially cheating by having the, um, you know, the h the hand - segmentations as the basis of the recognition.Professor F: Mm - hmm.PhD C: And so now with Thilo's segmenter working so well, I think we should {pause} consider doing a {disfmarker}PhD E: Mmm. So.Grad B: Come on.PhD C: uh, doing {disfmarker}PhD E: Yeah. We {disfmarker} But {disfmarker}Professor F: Y think {disfmarker} you think we should increase the error rate.PhD E: Anyway. Yeah.PhD C: Yeah.PhD E: Yeah.Professor F: Good.PhD C: Yeah.Professor F: Yeah.PhD E: That - that's what I wanted to do anyway,PhD C: Yeah.PhD E: so we should just get together and {disfmarker}PhD G: Yeah.PhD C: Yeah.PhD G: And even {disfmarker} The good thing is that since you, um, have high recall, {comment} even if you have low precision cuz you're over - generating, that's good because we could train noise models in the recognizer for these kinds of, uh, transients and things that come from the microphones,PhD C: Right.PhD E: Yeah. Yeah.PhD G: but {vocalsound} I know that if we run recognition unconstrained on a whole waveform, we do very poorly because we're {disfmarker} we're getting insertions in places what {disfmarker} that you may well be cutting out.PhD C: Well {disfmarker}PhD E: Yeah.Professor F: Mm - hmm.PhD G: So we do need some kind of pre - segmentation.PhD C: We should {disfmarker} we should consider doing some extra things, like, um, you know, retraining or adapting the {disfmarker} {vocalsound} the models for background noise to the {disfmarker} to this environment, for instance.PhD G: Mmm. Yeah.PhD E: Yeah.PhD G: And, yeah, using Thilo's, you know, posteriors or some kind of {disfmarker} or {disfmarker}PhD C: So.PhD G: right now they're {disfmarker} they're discrete,PhD E: Yeah.PhD G: yes or no for a speaker, to consider those particular speaker background models.PhD C: Right.PhD G: So. There's lots of ins interesting things that could be done.PhD E: Yeah. Yeah. We should do that.PhD G: So.Professor F: Good. So, uh, why don't we, uh, do the IBM stuffPhD D: Yeah. So, um, talked with Brian and gave him the alternatives to the single beep at the end of each utterance that we had generated before.Professor F: You had some {pause} thing about that Right.PhD D: And so {disfmarker}Professor F: The, uh, Chuck chunks.PhD D: Yeah. The Chuck chunks.PhD E: Hmm.PhD D: Right. And so he talked it over with the transcriber and the transcriber thought that the easiest thing for them would be if there was a beep and then the nu a number, a digit, and then a beep, uh, at the beginning of each oneProfessor F: Yeah. Yeah.PhD D: and that would help keep them from getting lost. And, um, {vocalsound} so Adam wrote a little script to generate those style, uh, beepsPhD C: Where'd you get the digits fromPhD D: and so we're {disfmarker} I came up here and just recorded the numbers one through ten.Postdoc A: They sound really good.PhD D: So. Does it sound OKPhD G: That's a great idea.Postdoc A: Yeah.PhD D: So, um {disfmarker} Yeah. We just used those.PhD C: And do you splice them into the {pause} waveform Or {disfmarker}PhD D: Yeah. He {disfmarker} then he d I recorded {disfmarker} Actually, I recorded one through ten three times at three different speeds and then he picked.PhD C: Right. Mm - hmm.PhD D: He liked the fastest one, so he just cut those out {vocalsound} and spliced them in between, uh, two beeps.Postdoc A: It sounds like a radio announcer's voice. Really.PhD E: It will be funny uh {disfmarker}Postdoc A: Yeah, yeah.PhD D: Does itPhD E: It will be funny when you're really reading digits, and then there are the chunks with {disfmarker} with your {pause} digits inPhD D: Yeah. With my {disfmarker}Postdoc A: Oh that's right.PhD G: Oh, right.PhD E: Yeah.Postdoc A: Now actually,PhD D: That'll throw them,Postdoc A: we're {disfmarker} Are we handling {disfmarker}PhD D: huhProfessor F: Uh, maybe we should have you record A, B, C for those or something.PhD D: Yeah. {comment} Huh! Maybe. And she said it wasn't gonna {disfmarker} the transcriber said it wouldn't be a problem cuz they can actually make a template, uh, that has beep, number, beep. So for them it'll be very quickPhD E: OK.PhD D: to {disfmarker} to put those in there {vocalsound} when they're transcribing.Professor F: Yeah.PhD D: So, um, we {disfmarker} We're gonna send them one more sample meeting, uh, and Thilo has run his segmentation. Adam's gonna generate the chunked file. And then, um, I'll give it to Brian and they can try that out. And when we get that back we'll see if that sort of fixes the problem we had with, uh, too many beeps in the last transcription.Professor F: OK. Do w do {disfmarker} what {disfmarker} Do you have any idea of the turn - around on {disfmarker} on those steps you just saidPhD G: Great.PhD D: Uh. Our s our {disfmarker} On our sideProfessor F: Uh.PhD D: or including IBM'sProfessor F: Including IBM's.PhD D: Well, I don't know. The last one seemed like it took a couple of weeks. Um, maybe even three.Professor F: OK.PhD D: Uh, that's just the I B M side. Our side is quick. I mean, I {disfmarker} I don't know. How long does your {disfmarker}PhD E: It should @ @ be finished today or something. Yeah.Professor F: Well, I meant the overall thing.PhD D: Yeah.Professor F: e e u u {comment} The reason I'm asking is because, uh, Jane and I have just been talking, and she's just been doing. {vocalsound} Uh, e a, you know, further hiring of transcribers.PhD D: Mm - hmm. Mm - hmm.Professor F: And so we don't sort of really know exactly what they'll be doing, how long they'll be doing it, and so forth, because right now she has no choice but to operate in the mode that we already have working.PhD D: Right.Professor F: And, uh, so it'd be {disfmarker} It'd be good to sort of get that resolved, uh, soon as we could,PhD D: Yeah. I {disfmarker} Yeah, I {disfmarker} I hope @ @ {comment} we can get a better estimate from this {pause} one that we send them.Professor F: and then {disfmarker}PhD D: So. Um.Professor F: Mm - hmm.PhD D: I {disfmarker} I don't know yet how long that'll take.Professor F: Yeah. Um {disfmarker} I mean in particular I would {disfmarker} I would really hope that when we do this DARPA meeting in July that we sort of have {disfmarker} we're {disfmarker} we're into production mode, somehow {disfmarker}PhD D: Mm - hmm.Professor F: You know, that we {disfmarker} we actually {vocalsound} have a stream going and we know how {disfmarker} how well it does and how {disfmarker} and how it operates.PhD D: Yeah.Professor F: I think that would {disfmarker} that would certainly be a {disfmarker} a very good thing to know.PhD D: Right. Right.Professor F: OK. Uh. Maybe before we do the meeting info organize thing, maybe you could say relevant stuff about where we are in transcriptions.Postdoc A: OK. So, um, we {disfmarker} {vocalsound} Uh, the transcribers have continued to work past what I'm calling" set one" , which was the s the set that I've been, uh {disfmarker} OK, talking about up to this point, but, uh, they've gotten five meetings done in that set. Right now they're in the process of being edited. Um, the, um {disfmarker} Let's see, I hired two transcribers today. I'm thinking of hiring another one, which will {disfmarker} because we've had a lot of attrition. And that will bring our total to {disfmarker}Professor F: They die off after they do this for a while.Postdoc A: Yeah. Well, you know, it's {disfmarker} it's various things.PhD D: Burn - out.Professor F: Yeah.Postdoc A: So, one of them had a baby. Um, you know, one of them really w wasn't planning {disfmarker}PhD C: Oh, that was an unfor unforeseen side effect of {disfmarker}Postdoc A: Eh, one of them, um, had never planned to work past January. I mean, it's th all these various things, cuz we, you know, we presented it as possibly a month project back in January and {disfmarker} and {disfmarker} and {disfmarker} and {disfmarker} Um, so it makes sense. Uh, through attrition we {disfmarker} we've {disfmarker} we're down to {disfmarker} to two, but they're really solid. We're really lucky the two that we kept. And, um {disfmarker} Well, I don't mean {disfmarker} I don't mean anything against the others. {comment} What I mean is we've got a good cause {disfmarker} a good core. No. We had a good core {disfmarker}PhD G: Well, they won't hear this since they're going. They won't be transcribing this meeting.Postdoc A: Yeah, but still. I mean, I d it's just a matter of we {disfmarker} w we're {disfmarker} we've got, uh,Professor F: No backs.Postdoc A: two of the ones who {disfmarker} who, um, ha had been putting in a lot of hours up to this point and they're continuing to put in a {disfmarker} a lot of hours, which is wonderful, and excellent work. And so, then, in addition, um, I hired two more today and I'm planning to h hire a third one with this {disfmarker} within this coming week, but {disfmarker} {vocalsound} but the plan is {disfmarker} just as, uh, Morgan was saying we discussed this, and the plan right now is to keep the staff on the {disfmarker} on the leaner side, you know, rather than hiring, like, eight to ten right now,Professor F: Mm - hmm.Postdoc A: because if the IBM thing comes through really quickly, then, um, we wouldn't wanna have to, uh, you know, lay people off and stuff. So. And this way it'll {disfmarker} I mean, I got really a lot of response for {disfmarker} for my notice and I think I could hire additional people if I {pause} wish to.Professor F: Yeah. An - and the other thing is, I mean, in the unlikely event {disfmarker} and since we're so far from this, it's a little hard to plan this way {disfmarker} in the unlikely event that we actually find {vocalsound} that we have, uh, transcribers on staff who are twiddling their thumbs because, you know, there's, you know, {vocalsound} all the stuff that {disfmarker} that was sitting there has been transcribed and they're {disfmarker} and they're faster {disfmarker} the {disfmarker} the pipeline is faster than {disfmarker} {vocalsound} uh, than the generation, um, eh, i in {disfmarker} in the day {disfmarker} e event that that day actually dawns, uh, I {disfmarker} I bet we could find some other stuff for them to do.Postdoc A: Oh, yes.Professor F: So I {disfmarker} I think {vocalsound} that, eh, eh, a as we were talking, if we {disfmarker} if we hire twelve, then we could, you know, run into a problem later. I mean, we also just couldn't sustain that forever. But {disfmarker} but, um {disfmarker} for all sorts of reasons {disfmarker} but if we hire f you know, f we have five on staff {disfmarker} five or six on staff at any given time, then {vocalsound} it's a small enough number so we can be flexible either way.Postdoc A: Good. OK.Professor F: Good.PhD G: It'd be great, too, if, um, we can {disfmarker} we might need some help again getting the tighter boundaries or some hand {disfmarker} to experiment with, um {disfmarker} {vocalsound} you know, to have a ground truth for this segmentation work, which {disfmarker} I guess you have some already that was really helpful, and we could probably use more.PhD E: Mmm, yeah. That was a thing I {disfmarker} I planned working on, is, uh, to use the {disfmarker} the transcriptions which are done by now, and to {disfmarker} to use them as, uh {disfmarker}PhD G: Oh. Oh, the new onesPhD E: Yeah.PhD G: with the tighter boundaries. Yeah.PhD E: Yeah. And to use them for {disfmarker} for training a {disfmarker} or for {disfmarker} fo whatever. Yeah. To {disfmarker} to create some speech - nonspeech labels out of them, and {disfmarker} Yeah, but that {disfmarker} that's a thing w was {disfmarker} w what I'm just looking into.PhD G: OK.Postdoc A: The {disfmarker} the {disfmarker} the pre - segmentations are so much {disfmarker} are s so extremely helpful. Now there was, uh, I g guess {disfmarker} So, a couple weeks ago I needed some new ones and it happened to be during the time that he was on vacation {disfmarker} f for just very few days you were away. But it happened to be during that time I needed one,PhD E: Yeah.Postdoc A: so I {disfmarker} {vocalsound} so I started them on the non - pre - segmented and then switched them over to yours and, um, they, um {disfmarker} you know, they always appreciate that when they have that available. And he's, uh, usually, eh, uh, um {disfmarker} Um. So they really appreciate it. But I was gonna say that they do adjust it once in a while. You know, once in a while there's something like,PhD E: Yeah, sure.Postdoc A: um, and e Actually you talked to them. Didn't you Did you Have you {disfmarker}PhD E: Yeah. I talked to Helen.Postdoc A: And {disfmarker} and {disfmarker} and she was {disfmarker} And so, I asked her {disfmarker} I mean, They're very perceptive. I really want to have this meeting of the transcribers. I haven't done it yet, but I wanna do that and she's out of town, um, for a couple of weeks, but I wanna do that when she returns. Um, cuz she was saying, you know, in a {disfmarker} in a span of very short period {disfmarker} we asked {disfmarker} It seems like the ones that need to be adjusted are these {disfmarker} these {disfmarker} these things, and she was saying the short utterances, uh, the, um {disfmarker}PhD G: Hmm.PhD E: Mmm. Yeah.Postdoc A: you know, I mean, you're {disfmarker} You're aware of this. But {disfmarker} but actually i it's so correct for so much of the time, that it's an enormous time saverPhD E: Yeah.Postdoc A: and it just gets tweaked a little around the boundaries. So.PhD G: That's great.Postdoc A: Um. Yeah. I think it'd be interesting to combine these.PhD E: Yeah.PhD G: Is there actually a record of where they change I mean, you can compare, do a diff on the {disfmarker} just so that we {vocalsound} knew {disfmarker}Postdoc A: You could do it. It's {disfmarker} it's complicated in that {vocalsound} um, {vocalsound} hhh, i hhh, iPhD E: Yeah. Actually, when {disfmarker} when they create new {disfmarker} yeah, new segments or something, it will be, uh, not that easy but {disfmarker} hmm. I think {pause} one could do that.PhD G: I mean, if we keep a old copy of the old time marksPhD E: Yeah.PhD G: just so that if we run it we know whether we're {disfmarker} which ones were cheatingPhD E: Yeah. Yeah. That would be great, yeah, to know that.PhD G: andPostdoc A: There is a {disfmarker} there is one problem with that and that is when they start part way through then what I do is I merge what they've done with the pre - segmented version.PhD G: which one would be good.PhD E: Yeah.Postdoc A: So it's not a pure {disfmarker} it's not a pure condition. Wha - what you'd really like is that they started with pre - segmented and were pre - segmented all the way through.PhD G: Mm - hmm.Postdoc A: And, um {disfmarker} @ @ {comment} {disfmarker} I, uh {disfmarker} the {disfmarker} it wasn't possible for about four of the recent ones. But, it will be possible in the futurePhD E: Yeah.Postdoc A: because we {disfmarker} we're, um.PhD E: It would.PhD G: Mmm, that's great.PhD E: Yeah.PhD G: Yeah. As long as we have a record, I guess, of the original {pause} automatic one, we can always find out how well {pause} we would do fr from the recognition side by using those boundaries.PhD E: Yeah. Yeah.PhD G: Um.PhD E: Yeah.PhD G: You know, a completely non - cheating version.PhD E: Yeah.PhD G: Also if you need someone to record this meeting, I mean, I'm happy to {disfmarker} for the transcribers {disfmarker} I could do it, or Chuck or Adam.Postdoc A: Thank you.Professor F: OK. So, uh, u you were saying something about organizing the meeting infoPhD D: Yeah. So, um, uh, Jane and Adam and I had a meeting where we talked about the reorganization of the {pause} directory structure for all of the meeting {disfmarker}Professor F: Did you record itPhD D: No. For all the Meeting Recorder data. We should have. Um. And so we've got a plan for what we're gonna do there. And then, Jane also s prepared a {disfmarker} um, started getting all of the {disfmarker} the meetings organized, so she prepared a {disfmarker} {vocalsound} a spreadsheet, which I spent the last couple of days adding to. So I went through all of the data that we have collected so far, and have been putting it into, uh, a spreadsheet {vocalsound} with start time, the date, the old meeting name, the new meeting name, the number of speakers, the duration of the meeting, comments, you know, what its transcription status is, all that kind of stuff. And so, the idea is that we can take this and then export it as HTML and put it on the Meeting Recorder web page so we can keep people updated about what's going on.PhD G: Oh, great.PhD D: Um, I've gotta get some more information from Jane cuz I have some {disfmarker} some gaps here that I need to get her to fill in, but {vocalsound} so far, um, {vocalsound} as of Monday, the fourteenth, um, we've had a total number of meeting sixty - two hours of meetings that we have collected. And, um {disfmarker} Uh, some other interesting things, average number of speakers per meeting is six. Um, and I'm gonna have on here the total amount that's been transcribed so far, but I've got a bunch of {disfmarker} uh, that's what I have to talk to Jane about, figuring out exactly which ones have {disfmarker} have been completed and so forth. But, um, {vocalsound} this'll be a nice thing that we can put up on the {disfmarker} the web site and people can {vocalsound} be informed of the status of various different ones. And {vocalsound} it'll also list, uh, like under the status, if it's at IBM or if it's at ICSI, uh, or if it's completed or which ones we're excluding and {disfmarker} and there's a place for comments, so we can, {vocalsound} um, say why we're excluding things and so forth. So.Professor F: Now would the ones that, um, are already transcribed {disfmarker} we h we have enough there that c you know, we've already done some studies and so forth and {disfmarker} um, shouldn't we go through and do the business - es u of {disfmarker} of having the, um, uh, participants approve it, uh, for {disfmarker} approve the transcriptions for distribution and so forthPostdoc A: Um, interesting idea. In principle, I {disfmarker} I would say yes, although I still am doing some {disfmarker} the final - pass editing, trying to convert it over to the master file as the {disfmarker} being the channelized version and it's {disfmarker} Yeah, it seems like I get into that a certain way and then something else intervenes {comment} and I have to stop. Cleaning up the things like the, uh, uh, places where the transcriber was uncertain, and {disfmarker} and doing spot - checking here and there. So, um, uh, I guess it would make sense to wait until th that's done, um, but {disfmarker} but {disfmarker}Professor F: Well, le let me put in another sort of a milestone kind of {disfmarker} as {disfmarker} as I did with the, uh, uh {disfmarker} the {disfmarker} the pipeline.Postdoc A: Yeah.Professor F: Um, we are gonna have this DARPA {pause} meeting in the middle of July,Postdoc A: Yes.Professor F: and I think it w it'd be {disfmarker} given that we've been {disfmarker} we've given a couple public talks about it already, spaced by months and months, I think it'd be pretty bad if we continued to say none of this is available. Um.Postdoc A: It'll certainly be done by then. Yeah.Professor F: Right. So we can s we {disfmarker} we wanna be able to say" here is a subset that is available right now" Postdoc A: Mm - hmm. That's right.Professor F: and that's has been through the legal issues and so forth.Postdoc A: That's right.Professor F: So.Postdoc A: Yeah. That's right. So that {disfmarker}Professor F: OKPostdoc A: OK.Professor F: So, by {disfmarker} before July.PhD C: And they don't have to approve, you know, th an edited version, they can just give their approval to whatever versionPostdoc A: Well, maybe {disfmarker}Professor F: Well, in principle, yes. But, I mean, i if {disfmarker} if {disfmarker} if somebody actually did get into some legal issue with it then wePhD C: Bu Yeah. But th I mean, the editing will continue. Presumably if {disfmarker} if s errors are found, they will be fixed, but they won't change the {disfmarker} the content of the meetings.PhD D: Content, really.Postdoc A: Well, see, this is the {disfmarker} this is the issue. Subtleties.PhD C: So.PhD G: Well, i if Jane is clarifying question question, then, you know, how can they agree to it before they know her final versionPostdoc A: The other thing, too, is there can be subtleties where a person uses this word instead of that word, which @ @ {comment} could've been transcribed in the other way.Professor F: Yeah.PhD G: Thing {disfmarker}Postdoc A: And no and they wouldn't have {vocalsound} been slanderous if it had been this other word. You knowProfessor F: I it {disfmarker} you know, there there is a point at which I agree it becomes ridiculous because, you know, you could do this final thing and then a year from now somebody could say, you know, that should be a period and not a question mark. Right And you don't {disfmarker} you {disfmarker} there's no way that we're gonna go back and ask everybody" do you approve this, uh, you know {disfmarker} this document now" So {disfmarker} So I think what it is is that the {disfmarker} the {disfmarker} the {disfmarker} the thing that they sign {disfmarker} I {disfmarker} I haven't looked at it in a while, but it has to be open enough that it sort of says" OK, from now on {disfmarker} you know, now that I've read this, you can use {disfmarker} do anything you want with these data." Postdoc A: Mm - hmm.Professor F: And, uh {disfmarker} But, i I think we wanna {disfmarker} So, assuming that it's in that kind of wording, which I don't remember, {vocalsound} um, I think i we just wanna have enough confidence ourselves that it's so close to the final form it's gonna be in, a year from now that they're {disfmarker}Postdoc A: Mm - hmm. I agree. Mmm. I totally agree. It's just, uh, a question of, {vocalsound} uh, if {disfmarker} if the person is using the transcript as the way of them judging what they said and whether it was slanderous, {vocalsound} then it seems like it's {disfmarker} it's {disfmarker} i it needs to be more correct than if we could count on them re - listening to the meeting.Professor F: Uh.Postdoc A: Because it becomes, eh, in a way a {disfmarker} a f uh, a legal document i if they've agreed to that.Professor F: Well, I forget how we end Right. I forget how we ended up on this, but I remember my taking the position {vocalsound} of not making it so {disfmarker} so easy for everybody to observe everything and Adam was taking the position of {disfmarker} of having it be really straightforward for people to check every aspect of it including the audio. And I don't remember who won, Adam or me, but {disfmarker}Postdoc A: Well, {vocalsound} if it's only the transcript, though {disfmarker} I mean, th this {disfmarker} this is my point, that {disfmarker} thatProfessor F: uh, the {disfmarker} Uh, that that's why I'm bringing this up again, because I can't remember how we ended up.Postdoc A: then it becomes {disfmarker}Professor F: That it was the transcrip He wanted to do a web interface that would make it {disfmarker}Postdoc A: Well, if it's just the audio {disfmarker} Well.Professor F: that would give you access to the transcript and the audio. That's what Adam wanted.Postdoc A: Mm - hmm.Professor F: And I don't remember how we ended up.PhD G: I mean, with the web interface it's interesting, because you could allow the person who signs to be informed when their transcript changes, or something like that. And, I mean, I would say" no" . Like, I don't wanna know, but some people might be really {vocalsound} interested and then y In other words, they would be informed if there was some significant change other than typos and things like that.Professor F: You decided you were whispering Satanic incantations under your breath when you were {disfmarker}PhD G: Well, {vocalsound} I don't know what happened to the small heads thing, but I j {vocalsound} Um, I'm just saying that, like, you know, you can sort of say that any things that are deemed {disfmarker}Professor F: They disappeared from view.PhD G: Anyway. I mean, I agree that at some point people {vocalsound} probably won't care about typos but they would care about significant meaning changes and then they could be asked for their consent, I guess, if {disfmarker} if those change. Cuz assumi {vocalsound} assuming we {disfmarker} we don't really distribute things that have any significant changes from what they sign anyway.PhD C: Tha That's {disfmarker} How about having them approve the audio and not the transcriptsPhD G: Oh, my God.Postdoc A: That would be simpler,Professor F: Uh.Postdoc A: if we could count on them listening.PhD G: But no one will listen to the hours and hours of {disfmarker}PhD D: Talk.PhD C: Well, that's O K.Grad B: That's {disfmarker}PhD C: We just have to give them a chance to listen to it, and if they don't, that's their problem.Grad B: hmm, hmm.PhD G: You {disfmarker} you d That's like {disfmarker}Postdoc A: Unfortunately, uh, in {disfmarker} in the sign thing that they signed, it says" transcripts" .PhD C: No, I'm serious.Postdoc A:" You'll be {disfmarker} you'll be provided the transcripts when they're available." PhD C: ReallyGrad B: Mmm.PhD C: Mmm.Professor F: Yeah.PhD E: Yeah.PhD G: I I {disfmarker} I thinkPostdoc A: Yeah.PhD G: that's a lot to ask for people that have been in a lot of meetings.Postdoc A: Yeah.Professor F: W anyway, haven't we {disfmarker} we've gone down this path a number of times. I know this can lead to extended conversations and {disfmarker} and not really get anywhere, so let {disfmarker} let me just suggest that {disfmarker} {vocalsound} uh, off - line that, uh, the people involved figure it out and take care of it before it's July.Postdoc A: Yes.Professor F: OK. So {disfmarker} so that in July we can tell people {vocalsound}" yes, we have this and you can use it" .Postdoc A: Yes. It's done, ready, available. Good.Professor F: Uh. So, let's see. What else we got Uh. Don did {disfmarker} did a report about his project in class and, uh {disfmarker} an oral and written {disfmarker} written version.PhD G: Well.Professor F: So that was stuff he was doing with you. Yeah.PhD G: I mean, it's {disfmarker} I guess one thing we're learning is that the amount {disfmarker} We have eight meetings there because we couldn't use the non - native {disfmarker} all non - native meetings and {vocalsound} it's, well, probably below threshold on enough data for us for the things we're looking at because the {vocalsound} prosodic features are {pause} very noisy and so you {disfmarker} you need a lot of data in order to model them. Um, so we're starting to see some patterns and we're hoping that maybe with, {vocalsound} I don't know, double or triple the data {disfmarker} with twenty meetings or so, that we would start to get better results. But we did find that some of the features that, I gue Jane would know about, that are expressing sort of the {vocalsound} distance of, um, {vocalsound} boundaries from peaks in the utterance and {vocalsound} some {pause} local, um, range {disfmarker} pitch range effects, like how close people are to their floor, are showing up in these classifiers, which are also being given some word features that are cheating, cuz they're true words. Um, so these are based on forced alignment. Word features like, um, word frequency and whether or not something's a backchannel and so forth. So, we're starting to see, I think, some interesting patterns.Professor F: So the dominant features, including everything, were those {disfmarker} those quasi - cheating things. Right Where these are {disfmarker}Grad B: Sometimes not.PhD G: I think it depends what you're looking at, a actually.Grad B: Yeah. Sometimes {pause} positions in sentences obviously, or in spurts, was helpful. I don't know if that's cheating, too.PhD G: Right. Um,PhD C: Spurts wouldn't be. RightPhD G: spurts is not cheating except that of course you know the real words,Grad B: Right.PhD G: but roughly speaking, the recognized words are gonna give you a similar type of position.Grad B: Right. Would they give you the same number of words, thoughProfessor F: Right.PhD G: It's either early or late.PhD C: NoPhD G: Not exactly, but iGrad B: But ra somewhatProfessor F: On the average.PhD G: Y yeah it should be. Well, we don't know and actually that's one of the things we're interested in doing, is a sort of {disfmarker}Professor F: Uh - huh.PhD C: Have you tried using just time, as opposed to number of wordsPhD G: So.Grad B: I think ti uh {disfmarker} Just p time position, like when the word startsPhD C: Yeah.Grad B: I don't know if that was in the {disfmarker}PhD C: Well, no, I mean t time {disfmarker} time position relative to the beginning of the spurt.PhD G: Eh {disfmarker} You know, uhGrad B: Start.PhD G: Yeah,Grad B: Yeah. There's all these things to do.PhD G: uh, we didn't try it, but it's sGrad B: Like, there's a lot of different features you could just pull out.PhD C: Yeah. I mean that wouldn't be cheating because you can detect pause {pause} pretty well within the time.Grad B: Right.PhD G: Right.Professor F: How about time position normalized by speakPhD G: And it depends on speaking rate {disfmarker}Professor F: Yeah. {vocalsound} Yeah.PhD G: speaking rate. Yeah.Grad B: Yeah.PhD G: Yeah. That's actually why I didn't use it at first.Professor F: Yeah.PhD C: Mm - hmm.PhD G: But we {disfmarker} one of the interesting things was I guess you reported on some te punctuation type {disfmarker}Grad B: Yeah.PhD G: finding sentence boundaries, finding disfluency boundaries, and then I had done some work on finding from the foreground speech whether or not someone was likely to interrupt, so where {disfmarker} you know, if I'm talking now and someone {disfmarker} and {disfmarker} and Andreas is about to interrupt me, is he gonna choose a certain place in my speech, either prosodically or word - based. And there the prosodic features actually showed up and a neat thing {disfmarker} even though the word features were available. And a neat thing there too is I tried some {disfmarker} {vocalsound} putting the speaker {disfmarker} So, I gave everybody {vocalsound} a short version of their name. So the real names are in there, which we couldn't use. Uh, we should use I Ds or something. And those don't show up. So that means that overall, um, it wasn't just modeling Morgan, or it wasn't just modeling a single person,Professor F: Mm - hmm.PhD G: um, but was sort of trying to, {vocalsound} uh, get a general idea {disfmarker} the model {disfmarker} the tree classifier was trying to find general locations that were applicable to different speakers, even though there are huge speaker effects. So. The {disfmarker} but the main limitation now is I {disfmarker} because we're only looking at things that happen every {vocalsound} ten words or every twenty words, we need more {disfmarker} more data and more data per speaker. So. It'd also be interesting to look at the EDU meetings because we did include meeting type as a feature, so whether you were in a r Meeting Recorder meeting or a Robustness meeting did matter {vocalsound} to {pause} interrupts because there are just fewer interrupts in the Robustness meetings.Grad B: Mm - hmm.PhD G: And so the classifier learns more about Morgan than it does about sort of the average person,Professor F: Mm - hmm.PhD G: which is {vocalsound} not bad. It'd probably do better than {disfmarker} {vocalsound} Um, but it wasn't generalizing.Professor F: Yeah.PhD G: So it's {disfmarker} And I think Don, um {disfmarker} Well, we have a long list of things he's starting to look at now over the summer, where we can {disfmarker} And he'll be able to report on more things {pause} in the future. But it was great that we could at least go from the {disfmarker} {vocalsound} you know, Jane's transcripts and the, {vocalsound} uh, recognizer output and get it {pause} to this point. And I think it's something Mari can probably use in her preliminary report {disfmarker} like," yeah, we're at the point where we're training these classifiers and we're just {vocalsound} reporting very preliminary but suggestive results that {vocalsound} some features, both word and pro prosodic, work." The other thing that was interesting to me is that the pitch features are better than in Switchboard. And I think that really is from the close - talking mikes, cuz the pitch processing that was done has much cleaner behavior than {disfmarker} than the Switchboard telephone bandwidth.PhD C: W wh wh wh Better in what sensePhD G: Um. Well, first of all, the pitch tracks are m have less, um, halvings and doublings than {disfmarker} than Switchboard and there's a lot less dropout, so if you ask how many regions where you would normally expect some vowels to be occurring {vocalsound} are completely devoid of pitch information,Professor F: Mm - hmm.PhD G: in other words the pitch tracker just didn't get a high enough probability of voicing for words {disfmarker} for {disfmarker} for, you know, five wordProfessor F: Hmm.PhD G: there are much fewer than in Switchboard. So the missing {disfmarker} {vocalsound} We had a big missing data problem in Switchboard and, so the features weren't as reliable cuz they were often just not available.PhD D: Could it have to do with the {disfmarker} the lower frequency cut - off on the SwitchboardPhD G: So that's actually good. Ma - maybe. I mean, the tele we had telephone bandwidth for Switchboard and we had the an annoying sort of telephone handset movement problem that I think may also affect it.PhD D: Hmm.PhD G: So we're just getting better signals in {disfmarker} in this data. Which is nice. So.Professor F: Yeah.PhD G: Anyway, Don's been doing a great job and we hope to continue with, um, Andreas's help and also some of Thilo's help on this,Professor F: Great.PhD E: YPhD G: to {disfmarker} to try to get a non - cheating version of how all this would work.PhD E: Yeah. Sure. Yeah.Professor F: Has {disfmarker} has, uh {disfmarker} We just {disfmarker} I think, just talked about this the other day, but h has {disfmarker} has anybody had a chance to try changing, uh, insertion penalty sort of things with the {disfmarker} with the, uh {disfmarker} {vocalsound} uh, using the tandem system input for the {disfmarker}PhD C: Oh, yeah. I tried that. It didn't, um, help dramatically. The {disfmarker}PhD D: Were they out of balance I didn't {disfmarker} I didn't notice.PhD C: There were a little {disfmarker} the relative number of {disfmarker} I think there were a higher number of deletions, actually.Professor F: Oh.PhD C: So, you, uh {disfmarker} So, actually it {disfmarker} it preferred to have a positive {disfmarker} er, negative insertion penalty,PhD G: DeletionsPhD C: which means {vocalsound} that, um {disfmarker}Professor F: Uh - huh.PhD C: But, you know, it didn't change {vocalsound} th the {disfmarker} by adjusting that {disfmarker} the, um {disfmarker}Professor F: OK.PhD C: Yeah. The error changed by probably one percent or so. But, you know, given that that word error rate is so high, that's not a {disfmarker}Professor F: OK. So that {disfmarker} So that's {disfmarker} So that's not the problem.PhD C: That's not the problem. No.Professor F: Yeah.PhD C: But, uh, we s just, um, uh {disfmarker} you know, Chuck and I talked and the {pause} @ @ {comment} next thing to do is probably to tune the {disfmarker} um, the size of the Gaussian system, um, @ @ {comment} to {disfmarker} to this {disfmarker} to this feature vector, which we haven't done at all. We just used the same {vocalsound} configuration as we used for the {disfmarker} {vocalsound} for the standard system.Professor F: Hmm.PhD C: And, {vocalsound} for instance, uh, Dan {disfmarker} @ @ {comment} Dan just sent me a message saying that CMU used, um, {vocalsound} something like ten Gaussians per cluster {disfmarker} You know, each {disfmarker} each mixture has ten {pause} GaussiansPhD D: Mm - hmm. Hmm. We're using sixty - four,PhD C: and {disfmarker} and we're using sixty - four,PhD D: rightPhD C: so that's {vocalsound} obviously a big differencePhD D: Yeah.PhD C: and it might be way off and give very poorly trained, uh, you know, Gaussians that way,Professor F: Hmm.PhD C: uh, an and poorly trained mixture weights. So {disfmarker} so, we have {disfmarker} The turn - around time on the training when we train only the {disfmarker} a male system with, uh, you know, our small training set, is {vocalsound} less than twenty - four hours, so we can run lots of {disfmarker} {vocalsound} uh, basically just brute force, try a whole bunch of different um, settings.Professor F: OK.PhD C: And, uh, with the new machines it'll be even better. So.Professor F: Yeah. We get twelve of those,PhD C: Yeah.Professor F: huhPhD C: But the PLP features work {disfmarker} um, uh, you know, continue to improve the,Professor F: OK.PhD C: um {disfmarker} As I said before, the {disfmarker} uh using Dan's, uh, uh, vocal tract normalization option works very well. So, um, @ @ {comment} I ran one experiment where we're just {vocalsound} did the vocal tract le normalization only in the test data,Professor F: Mm - hmm.PhD C: so I didn't bother to retrain {pause} the models at all, and it improved by one percent, which is about what we get with {disfmarker} {vocalsound} uh, with, you know, just @ @ {comment} actually doing both training and test normalization, um, with, um, {vocalsound} the, uh {disfmarker} {vocalsound} uh, with the standard system. So, in a few hours we'll have the numbers for the {disfmarker} for retraining everything with vocal tract length normalization and {disfmarker} So, that might even improve it further.Professor F: Great.PhD C: So, it looks like the P L - fea P features {comment} do very well now with {disfmarker} after having figured out all these little tricks to {disfmarker} to get it to work.Professor F: Yeah.PhD C: So.Professor F: Good.PhD G: Wait. So you mean you improve one percent over a system that doesn't have any V T L in it alreadyPhD C: Exactly. Yeah.PhD G: OK.Professor F: Yeah. OK. So then {disfmarker} then we'll have our baseline to {disfmarker} to compare the currently hideous, uh, uh, new thing with.PhD C: Right. a Right. And {disfmarker} and what that suggests also is of course that the current Switchboard {pause} MLP isn't trained on very good features.Professor F: But {disfmarker} Yeah.PhD C: Uh, because it was trained on whatever, you know, was used, uh, last time you did Hub - five stuff, which didn't have any of the {disfmarker}Professor F: Right. But all of these effects were j like a couple percent.PhD C: Uh.Professor F: Right I mean, y the {disfmarker}PhD C: Well, but if you add them all up you have, uh, almost five percent difference now.Professor F: Add {pause} all of them. I thought one was one point five percent and one was point eight.PhD C: Yeah. And now we have another percent with the V TProfessor F: That's three point three.PhD C: Um, actually, and it's, um, What's actually qu interesting is that with {disfmarker} um, well, you m prob maybe another half percent if you do the VTL in training, and then interestingly, if you optimize you get more of a win out of rescoring the, um, {vocalsound} uh, the N best lists, uh, and optimizing the weights, um, uh than {disfmarker}PhD D: Than you do with the standardPhD C: Yeah. So {disfmarker}Professor F: Yeah. But the part that's actually adjustment of the front - end per se as opposed to doing {disfmarker} putting VTLN in or something is {disfmarker} it was a couple percent.PhD C: Right.Professor F: Right It was {disfmarker} it was {disfmarker} there was {disfmarker} {vocalsound} there was one thing that was one and a half percent and one that was point eight. So {disfmarker} and {disfmarker} and {disfmarker} let me see if I remember what they were. One of them {vocalsound} was, uh, the change to, uh {disfmarker} because it did it all at once, {comment} to {disfmarker} uh, from bark scale to mel scale,PhD C: Mm - hmm.Professor F: which I really feel like saying in quotes, because @ @ {comment} they're essentially the same scale but the {disfmarker} but {disfmarker} but {disfmarker} but any i individual particular implementation of those things puts things in a particular place.PhD G: Yeah. Why did that chaPhD C: Mm - hmm.Professor F: So that's why I wanted to look {disfmarker} I still haven't looked at it yet. I {disfmarker} I wanna look at exactly where the filters were in the two,PhD C: Mm - hmm.Professor F: and it {disfmarker} {vocalsound} it's probably something like there's one fewer or one more filter in the sub {vocalsound} one kilohertz bandPhD C: Mm - hmm.Professor F: and for whatever reason with this particular experiment it was better one way or the other.PhD G: Hmm.Professor F: Um, it could be there's something more fundamental but it {disfmarker} you know, I {disfmarker} I don't know it yet. And the other {disfmarker} and the other {disfmarker} that was like one and a half or something, and then there was point eight percent, which was {disfmarker} what was the other thingPhD D: Well, that was combined with the triangular. RightProfessor F: Yeah. Those {disfmarker} those two were together.PhD D: Yeah. Right.Professor F: We d weren't able to separate them out cuz it was just done in one thing. But then there was a point eight percent which was something else.PhD D: The low - frequency cut - off.Professor F: Do you remember the {disfmarker} Oh, yeah. So that was {disfmarker} that was, uh {disfmarker} that one I can claim credit for, uh, i in terms of screwing it up in the first place. So that someone e until someone else fixed it, which is that, um, I never put {disfmarker} when I u We had some problems before with offsets. This inf this went back to, uh, I think Wall Street Journal.PhD C: Hmm.Professor F: So we {disfmarker} we had, uh {disfmarker} ea everybody else who was doing Wall Street Journal knew that there were big DC offsets in th in these data {disfmarker} in those data and {disfmarker} and {disfmarker} and nobody happened to mention it to us,PhD C: Hmm.Professor F: and we were getting these, like, really terrible results, like two, three times the error everybody else was getting. And then in casual conversation someone ment mentioned" uh, well, I guess, you know, of course you're taking care of the offsets." I said" what offsets" Grad B:PhD C: Mm - hmm.Professor F: And at that point, you know, we were pretty new to the data and we'd never really, like, looked at it on a screen and then when we just put it on the screen {comment} and wroop!PhD C: Mm - hmm.Professor F: there's this big DC offset. So, um, in PLPPhD G: There was a {disfmarker} like a hum or some or {disfmarker} when they recorded itProfessor F: No. It's just, it {disfmarker} it's {disfmarker} it's not at all uncommon for {disfmarker} for recorded electronics to have different, um, DC offsets.PhD G: Or just {disfmarker} Huh.Professor F: It's {disfmarker} it's, you know, no big deal. It's {disfmarker} you know, you could have ten, twenty, maybe thirty millivolts, whatever, and it's consistently in there. The thing is, most people's front - ends have pre - emphasis with it, with zero at zero frequency, so that it's irrelevant. Uh, but with P L P, we didn't actually have that. We had {disfmarker} we had the equivalent of pre - emphasis in a {disfmarker} a, uh, Fletcher - Munson style weighting that occurs in the middle of P L but it doesn't actually have a zero at zero frequency,PhD G: Hmm.Professor F: like, eh, uh, typical simple fr pre - emphasis does. We had something more fancy. It was later on it didn't have that. So at that point I reali" oh sh we better have a {disfmarker} have a high - pass filter" just, you know {disfmarker} {vocalsound} just take care of the problem. So I put in a high - pass filter at, uh, I think ninety {disfmarker} ninety hertz or so {vocalsound} uh, for a sixteen kilohertz sampling rate. And I never put anything in to adjust it for different {disfmarker} different sampling rates. And so {disfmarker} well, so, you know, the code doesn't know anything about that and so this is all at eight kilohertz and so it was at forty - five hertz instead of at {disfmarker} {vocalsound} instead of at ninety.PhD C: Hmm.Professor F: So, um, I don't know if Dan fixed it or {disfmarker} or, uh, what he {disfmarker}PhD C: Well, he made it a parameter.Professor F: He made it a parameter. So. Yeah, I guess if he did it right, he did fix it and then {disfmarker} and then it's taking care of sampling rate, which is great.PhD D: What {disfmarker} what is the parameterProfessor F: He had a {disfmarker}PhD D: Is it, uh, just the f lower cut - off that you wantPhD C: It's called, uh, {vocalsound} H - HPF.Professor F: H {disfmarker} Yeah. Does HPF on {disfmarker} on his feat feature.PhD C: u And {disfmarker} but HPF, you know, when you put a number after it, uses that as the hertz value of the cut - off.PhD D: Mm - hmm. Oh, OK.Professor F: Yeah.PhD C: So.Professor F: I mean, frankly, we never did that with the RASTA filter either,PhD C: Mm - hmm.Professor F: so the RASTA filter is actually doing a different thing in the modulation spectral domain depending on what sampling rate you're doing, which is {vocalsound} another old {disfmarker} old bug of mine.PhD C: Mm - hmm.Professor F: But, um {disfmarker} Um. So that {disfmarker} that was the problem there was th we {disfmarker} we {disfmarker} we had always intended to cut off below a hundred hertzPhD C: Mm - hmm.Professor F: and it just wasn't doing it, so now it is. So, {vocalsound} that hep that helped us by, like, eight tenths of a percent. It {pause} still wasn't a big deal.PhD C: OK. Well, but, um {disfmarker} Well, uh, again, after completing the {pause} current experiments, we'll {disfmarker} {vocalsound} we can add up all the uh differencesProfessor F: Oh, yeah.PhD C: and {disfmarker} {vocalsound} and {disfmarker} anProfessor F: But {disfmarker} but, I guess my {disfmarker} my point was that {disfmarker} that, um, the hybrid system thing that we did was, uh, primitive in many ways.PhD C: Y Right.Professor F: And I think I agree with you that if we fixed lots of different things and they would all add up, we would probably have a {disfmarker} a {disfmarker} a competitive system. But I think not that much of it is due to the front - end per se. I think maybe a couple percent of it is, as far as I can see from this.PhD C: Mm - hmm.Professor F: Uh, unless you call {disfmarker} well, if you call VTL the front - en front - end, that's, uh, a little more. But that's sort of more both, kind of.PhD D: One experiment we should {disfmarker} we'll probably need to do though when {disfmarker} um, at some point, is, since we're using that same {disfmarker} the net that was trained on PLP without all these things in it, for the tandem system, we may wanna go back and retrain,Professor F: Right But.PhD C: Well, that's what I meant, in fact. Yeah.PhD D: yeah, yeah, for the tandem. You know, so we can see if it {disfmarker} what effect it has on the tandem processing.PhD C: So {disfmarker} so, the thing is {disfmarker} is do we expect {disfmarker}Professor F: Mm - hmm.PhD C: eh At this point I'm as I mean, you know {disfmarker} e I'm wondering is it {disfmarker} Can we expect, uh, a tandem system to do better than a properly trained {disfmarker} you know, a Gaussian system trained directly on the features with, you know, the right ch choice of {pause} parametersProfessor F: Well, that's what we're seeing in other areas. Yes. Right So, it's {disfmarker} so, um, um {disfmarker}PhD D: So, we {disfmarker} But {disfmarker} but we may not. I mean, if it doesn't perform as well, we may not know why. Right Cuz we need to do the exact experiment.PhD C: Right.Professor F: I mean, the reason to think it should is because you're putting in the same information and you're transforming it to be more discriminative. So. Um. Now the thing is, in some databases I wouldn't expect it to necessarily give you much and {disfmarker} and part of what I view as the real power of it is that it {pause} gives you a transformational capability {vocalsound} for taking all sorts of different wild things that we do, not just th the standard front - end, but other things, like with multiple streams and so forth,PhD C: Mm - hmm.Professor F: and allows you to feed them to the other system with this {disfmarker} through this funnel. Um, so I think {disfmarker} I think that's the real power of it. I wouldn't expect huge in huge improvements. Um, but it should at least be roughly the same and maybe a little better.PhD C: Mm - hmm.Professor F: If it's, you know, like way way worse then, you know {disfmarker}PhD C: Right.PhD D: So, Morgan, an another thing that Andreas and I were talking about was, so @ @ {comment} in the first experiment that he did {vocalsound} we just took the whole fifty - six, uh, outputs and that's, um, basically compared to a thirty - nine input feature vector from either MFCC or PLP.Professor F: Mm - hmm. Mm - hmm.PhD D: But one thing we could do is {disfmarker}Professor F: Let {disfmarker} let me {disfmarker} let me just ask you something. When you say take the fifty - six outputs, these are the pre final nonlinearity {pause} outputsPhD D: Yeah. Through the regular tandem outputs.Professor F: and they're {disfmarker} and {disfmarker} through the KLT.PhD D: Through the KLT. All that kinda stuff.Professor F: OK. And so {disfmarker} so then you u Do you use all fifty - six of the KLTPhD D: That's what we did.Professor F: or {disfmarker}PhD D: Right So one thing we were wondering is, if we did principal components and, say, took out just thirteen, and then did deltas and double - deltas on that {disfmarker}Professor F: OK. Yes.PhD D: so we treated the th first thirteen as though they were {vocalsound} standard features.Professor F: Yes. Yeah.PhD D: I mean, did Dan do experiments like that to {disfmarker}Professor F: Uh. Talk with Stephane. He did some things like that. It was either him or Carmen. I forget.PhD C: Mm - hmm.PhD D: Mmm.Professor F: I mean these were all different databases and different {disfmarker} you know, in HTK and all that,PhD D: Yeah.Professor F: so i it {disfmarker} it may not apply. But my recollection of it was that it didn't make it better but it didn't make it worse.PhD D: Hmm.Professor F: But, again, given all these differences, maybe it's more important in your case that you not take a lot of these low - variance, uh, components.PhD D: Cuz in a sense, the net's already got quite a bit of context in those features,Professor F: Yeah.PhD D: so if we did deltas and double - deltas on top of those, we're getting sort of even more.Professor F: Which could be good or not.PhD D: Yeah.Professor F: Yeah. Yeah. Worth trying.PhD C: But there the main point is that, um, you know, it took us a while but we have the procedure for coupling the two systems {vocalsound} debugged now and {disfmarker} I mean, there's still conceivably some bug somewhere in the way we're feeding the tandem features {disfmarker} uh, either generating them or feeding them to this {disfmarker} to the {vocalsound} SRI system,PhD D: Mm - hmm. Yeah.PhD C: but it's {disfmarker}Professor F: There might be, cuz that's a pretty big difference.PhD C: Yeah. And I'm wondering how we can {disfmarker} how we can debug that.Professor F: ButPhD D: Yeah.PhD C: I mean how {disfmarker} Um.Professor F: Hmm.PhD C: I'm actually f quite sure that the {disfmarker} feeding the {pause} features into the system and training it up,Professor F: What if {disfmarker}PhD C: that {disfmarker} that {disfmarker} I think that's {disfmarker} this {disfmarker} that's essentially the same as we use with the ce with the P L P fe features. And that's obviously working great. So. I um.PhD D: Yeah. There could be a bug in {disfmarker} in the {disfmarker} somewhere before that.PhD C: There {disfmarker} we could {disfmarker} the {disfmarker} another degree of freedom is how do you generate the K L T transformPhD D: Mm - hmm.PhD C: Right We toProfessor F: That's {disfmarker}PhD D: Right.Professor F: well, and another one is the normalization of the inputs to the net.PhD C: Yeah.Professor F: These nets are trained with particular normalization and when that gets screwed up it {disfmarker} it can really hurt it.PhD D: I'm doing what Eric {disfmarker} E Eric coached me through then {disfmarker} that part of it, so I'm pretty confident in that.Professor F: OK.PhD D: I mean, the only slight difference is that I use normalization values that, um, Andreas calculated from the original {comment} PLP,PhD C: Right.PhD D: which is right.PhD C: Right.PhD D: N Yeah. So, I u I do {disfmarker} Oh, we actually don't do that normalization for the PLP, do we For the st just the straight PLP featuresPhD C: No. The {disfmarker} the SRI system does it.PhD D: S R I system does that. Right.PhD C: Yeah.Professor F: Right. Well, you might e ePhD C: So, there's {disfmarker} there is {disfmarker} there is room for bugs that we might not have discovered,PhD D: So that's {disfmarker} that's another {disfmarker} Yeah.Professor F: Yeah.PhD D: Mm - hmm.Professor F: Yeah. {vocalsound} I {disfmarker} I would actually double check with Stephane at this point,PhD C: but {disfmarker}Professor F: cuz he's probably the one here {disfmarker} I mean, he and Dan are the ones who are at this point most experienced with the tandemPhD D: Mm - hmm.Professor F: thing and there may {disfmarker} there may be some little bit here and there that is not {disfmarker} not being handled right.PhD D: Yeah. It's hard with features, cuz you don't know what they should look like. I mean, you can't just, like, print the {disfmarker} the values out in ASCII and, you know, look at them, see if they're {disfmarker}Professor F: Not unless you had a lot of timePhD G: Well {disfmarker}Professor F: and {disfmarker}PhD G: eh, and also they're not {disfmarker} I mean, as I understand it, you {disfmarker} you don't have a way to optimize the features for the final word error. RightPhD C: Right.PhD G: I mean, these are just discriminative, but they're not, um, optimized for the final {disfmarker}PhD C: They're optimized for phone discrimination, not for {disfmarker}PhD G: Right. So it {disfmarker} there's always this question of whether you might do better with those features if there was a way to train it for the word error metric that you're actually {disfmarker} that you're actually {disfmarker}Professor F: That's right. Well, the other {disfmarker} Yeah, th the {disfmarker}PhD C: Mm - Mmm.Professor F: Well, you actually are. But {disfmarker} but it {disfmarker} but in an indirect way.PhD G: Well, right. It's indirect, so you don't know {disfmarker}Professor F: So wha w what {disfmarker} an and you may not be in this case, come to think of it, because, uh, you're just taking something that's trained up elsewhere. So, what {disfmarker} what you {disfmarker} what you do in the full procedure {vocalsound} is you, um, uh, have an embedded training. So in fact you {disfmarker} the {disfmarker} the net is trained on, uh, uh, a, uh, Viterbi alignment of the training data that comes from your full system. And so that's where the feedback comes all around, so that it is actually discriminant. You can prove that it's {disfmarker} it's a, uh {disfmarker} If you believe in the Viterbi assumption that, uh, getting the best path, uh, is almost equivalent to getting the best, uh, total probability, um, then you actually do improve that by, uh {disfmarker} by training up on local {disfmarker} local, uh {disfmarker} local frames. But, um, we aren't actually doing that here, because we did {disfmarker} we did that for a hybrid system, and now we're plugging it into another system and so it isn't {disfmarker} {vocalsound} i i i it wouldn't quite apply here.PhD C: Do yPhD D: So another huge experiment we could do would be to take the tandem features, uh, do SRI forced alignments using those features, and then re - do the net with those.Professor F: Mm - hmm.PhD G: Mmm, uh {disfmarker} Exactly. Exactly.Professor F: Yeah.PhD G: So that you can optimize it for the word error.PhD C: But {disfmarker}Professor F: Yeah. Another thing is since you're not using the net for recognition per se but just for this transformation, it's probably bigger than it needs to be.PhD G: Yeah.Professor F: So that would save a lot of time.PhD C: And there's a mismatch in the phone sets. So, you're using a l a long a larger phone set than what {disfmarker}PhD D: Mmm.Professor F: Yeah. Actually all those things could {disfmarker} could {disfmarker} could {disfmarker} could, uh {disfmarker} could affect it as well.PhD D: Yeah. Yeah.Professor F: The other thing, uh, just to mention that Stephane {disfmarker} this was an innovation of Stephane's, which was a pretty neat one, uh, and might particularly apply {vocalsound} here, given all these things we're mentioning. Um, Stephane's idea was that, um, discriminant, uh, approaches are great. Even the local ones, given, you know, these potential outer loops which, you know, you can convince yourself turn into the global ones. Um, however, there's times when it {pause} is not good. Uh, when {pause} something about the test set is different enough from the training set that {disfmarker} that, uh, the discrimination that you're learning is {disfmarker} is {disfmarker} is not a good one.PhD C: Mm - hmm.Professor F: So, uh, his idea was to take as the input feature vector to the, uh, Gaussian mixture system, {vocalsound} uh, a concatenation of the neural net outputs and the regular features.PhD C: Oh, we already talked about that.PhD G: Yeah. That {disfmarker}Professor F: Yeah.PhD C: ElPhD D: Mm - hmm.PhD G: Didn't you {disfmarker} did you {pause} do that alreadyPhD C: Yeah. No, but we {disfmarker} we {disfmarker} when {disfmarker} when we {disfmarker} when I first started corresponding with Dan about how to go about this, I think that was one of the things that we definitely went there.PhD G: or {disfmarker} Oh. That makes a lot of sense. Huh.Professor F: Yeah. Yeah. I mean, I'm sure that Stephane wasn't the first to think of it,PhD C: Yeah.Professor F: but actually Stephane did itPhD C: Uh - huh. And i does it helpProfessor F: and {disfmarker} {vocalsound} and {disfmarker} and it helped a lot.PhD C: Oh, OK.Professor F: Yeah. So that's {disfmarker} that {disfmarker} that's our current best {disfmarker} best system in the, uh {disfmarker} {vocalsound} uh, in the Aurora thing.PhD C: Oh. OK.PhD G: Yeah. That makes sense.PhD C: And do you do a KLT transform on the con on the combined feature vectorProfessor F: Yeah.PhD G: As {disfmarker} you should never do worse.Professor F: I {disfmarker} I, uh, missed what you said.PhD C: Do you {disfmarker} d you do a KLT transform on the combined feature vectorProfessor F: Yeah.PhD C: OK.Professor F: Well, actually, I, uh {disfmarker} you should check with him, because he tried several different combinations.PhD C: Because you end up with this huge feature vector, so that might be a problem, a unless you do some form of dimensionality reduction.Professor F: Yeah. I, uh, th what I don't remember is which came out best. So he did one where he put o put e the whole thing into one KLT, and another one, since the {disfmarker} the PLP things are already orthogonalized, he left them alone and {disfmarker} and just did a KLT on the {disfmarker} on the {disfmarker} on the net outputsPhD C: Mm - hmm. Mmm.Professor F: and then concatenated that. And I don't remember which was better.PhD D: Did he {disfmarker} did he try to {disfmarker} So he always ended up with a feature vector that was {pause} twice as long as either one of the {disfmarker}Professor F: No. I don't know, i I {disfmarker} I don't know. You have to check with him.PhD D: Yeah.PhD C: OK. Actually, I have to run.Professor F: I'm into big ideas these days.PhD G: Yeah.PhD C: Uh.PhD G: We need to close up cuz I need to save the data and, um, get a call.Professor F: Not to mention the fact that we're missing snacks. Yeah.PhD G: Right.Professor F: UhPhD G: Did people wanna do the digitsProfessor F: Um.PhD G: or, um, do them togetherProfessor F: I {disfmarker} I g I think, given that we're in a hurry for snacks, maybe we should do them together.PhD G: I don't know. Should we just {disfmarker} OK. I mean, are we trying to do them {nonvocalsound} in synchrony That might be fun.Professor F: Well, it's {disfmarker} {vocalsound} it's {disfmarker} it's not {disfmarker} You know, it's not gonna work outPhD G: Adam's not here, so he's not here to tell me no.Professor F: but we could {disfmarker} we could just, uh, uh, see if we find a rhythm, you know, what {disfmarker}PhD G: Sure.Professor F: Uh, O's or zeroes, we wanna agree on thatPhD G: Maybe just whatever people would naturally do I don't know.Professor F: Oh, but if we were a singing group, we would wanna decide. RightPhD G: Be harmony. Yeah. {comment} Yeah.Postdoc A: Mine's identical to yours.Professor F: We might waPostdoc A: Is that correctPhD G: Sorry. So I set up and we didn't have {pause} enough digit formsPostdoc A: Oh. I see.Professor F: So these are excellent.PhD G: so I xeroxed the same one seven times.Postdoc A: Oh. I see.Professor F: Why don't we do zer i Anyone have a problem with saying zero Is zero OKPhD G: No.Postdoc A: Yeah.Professor F: OK. One and a two and three.PhD G: eProfessor F: Once more with feeling.PhD G: And thProfessor F: No, just k just kidding. Oh, yeah. It was.
Segmentation for the recogniser has been done by hand which the group consider "cheating", instead now they want to use Thilo's automatic segmenter.
What change has occurred to the segmenter?