Q:

Solr custom filter TokenStream issue

For indexing and querying I need to perform certain transformations as listed below. Therefore I wrote a custom filter. How can I perform the concatenation of the token and it get passed to NGramFilterFactory filter. Please tell me what needs to be improved in the code.
This is the config from schema.xml file:
<tokenizer class="solr.WhitespaceTokenizerFactory"/>

    <filter class="solr.LowerCaseFilterFactory" />
    <filter class="solr.TrimFilterFactory" />
    <filter class="solr.TrimFilterFactory" pattern="([^a-z])" replacement="" replace="all" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="custom_stop_words.txt"/>
    <filter class="intuit.ripple.solr.ConcatFilterFactory" />
    <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="3" />

Here is an example of the use case I am trying to solve:
1. Input value: "foo Bar Baz qux"
2. WhitespaceTokenizerFactory: "foo", "Bar", "Baz", "qux"
3. LowerCaseFilterFactory: "foo", "bar", "baz", "qux"
4. TrimFilterFactory, TrimFilterFactory and StopFilterFactory have nothing to do in this case.
5. ConcatFilterFactory: "foobarbazqux". It should concatenate the tokens.
6. NGramFilterFactory: This will generate the token.

Here is the incrementToken() method from ConcatFilter:
@Override
public boolean incrementToken()  throws IOException  {

    StringBuilder builder = new StringBuilder();

    while (input.incrementToken()) {
        int len = charTermAtt.length();
        char buffer[] = charTermAtt.buffer();
        builder.append(buffer, 0, len);
        System.out.println("Tokens: " + new String(buffer, 0, len));
        clearAttributes();
        charTermAtt.setEmpty();
    }
    System.out.println("Concat tokens: " + builder.toString());

    charTermAtt.copyBuffer(builder.toString().toCharArray(), 0, builder.length());
    charTermAtt.setLength(builder.length());
    posIncAtt.setPositionIncrement(1);
    setOffsetAttr.setOffset(0, builder.length());

    input.end();
    input.close();
    return false;
}

Here I am using while loop to fetch all the token and joining them together. Is there any way to fetch all the token at once without looping?

A:

I think you wanted to do something else than you implemented :D
What you do in incrementToken method is just iterating through whole input (in this case output from StopFilter). In each call of increment token you should just take single (or more if needed) token from input and you produce SINGLE token to output. 
So I guess you didn't want to use "while" loop here and moreover call "clearAttributes()" in each interation. 
I also guess you have ouput like this:
Tokens: foo
Tokens: bar
Concat tokens: foo bar

But in fact from two tokens "foo" and "bar" you produced single token "foo bar" which I guess was not your intention. Please describe what your ConcatFilterFactory should do. Currently it is only merging multiple tokens to single one.
You have an example of discussion about TokenFilter here: http://search-lucene.com/m/ukJmjphJte/tokenfilter&subj=custom+TokenFilter. You can use this search box to find more information about Solr/Lucene related information: http://search-lucene.com/

