Task_type parameter of LoraConfig
I am training a fine-tune of codellama using PEFT but not sure how to use the task_type parameter of LoraConfig. Should it be CAUSAL_LM or SEQ_2_SEQ_LM or something else? Does it have any affect?The goal of my model is to parse an input for independent clauses in a sentence. For example, it would insert a delimiter, such as  in this sentence: “the tea  was on the stove  and was at high temperature” , separating the independent clause from the subordinate clause. My training data is all in a single col and each row looks like this (where the → and  are custom tokens I add to the tokenizer vocab and the  is the EOS token):“the tea  was on the stove and was at high temperature → the tea  was on the stove  and was at high temperature ”
2023-09-11T16:06:09Z
Does task_type matters in LoraConfig, and if so, in what way?
2023-09-17T22:05:07Z
I have the same question. Does task_type matter?
2023-10-17T08:54:44Z
In the source code,task_typeis not even there:github.comhuggingface/peft/blob/15a013af5ff5660b9377af24d3eee358213d72d4/src/peft/tuners/lora/config.py#L24# limitations under the License.from dataclasses import dataclass, fieldfrom typing import List, Optional, Unionfrom peft.config import PeftConfigfrom peft.utils import PeftType@dataclassclass LoraConfig(PeftConfig):"""This is the configuration class to store the configuration of a [`LoraModel`].Args:r (`int`): Lora attention dimension.target_modules (`Union[List[str],str]`): The names of the modules to apply Lora to.lora_alpha (`int`): The alpha parameter for Lora scaling.lora_dropout (`float`): The dropout probability for Lora layers.fan_in_fan_out (`bool`): Set this to True if the layer to replace stores weight like (fan_in, fan_out).For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set
2023-10-20T03:11:15Z
Thetask_typeparameter is used in the superclassPeftConfig
2024-02-25T18:02:17Z
SEQ_CLS = "SEQ_CLS"
    SEQ_2_SEQ_LM = "SEQ_2_SEQ_LM"
    CAUSAL_LM = "CAUSAL_LM"
    TOKEN_CLS = "TOKEN_CLS"
    QUESTION_ANS = "QUESTION_ANS"
    FEATURE_EXTRACTION = "FEATURE_EXTRACTION"Overview of the supported task types:
    - SEQ_CLS: Text classification.
    - SEQ_2_SEQ_LM: Sequence-to-sequence language modeling.
    - Causal LM: Causal language modeling.
    - TOKEN_CLS: Token classification.
    - QUESTION_ANS: Question answering.
    - FEATURE_EXTRACTION: Feature extraction. Provides the hidden states which can be used as embeddings or features
      for downstream tasks.See here for source:peft/src/peft/utils/peft_types.py at v0.8.2 · huggingface/peft · GitHub