Update README.md

README.md

@@ -1,199 +1,170 @@
 ---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
 library_name: transformers
-tags: []
 ---
 
-# Model Card for Model ID
+# Granite Uncertainty 3.0 8b
 
-<!-- Provide a quick summary of what the model is/does. -->
 
 
+## Model Summary
 
-## Model Details
+**Granite Uncertainty 3.0 8b** is a LoRA adapter for [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
+adding the capability to provide calibrated certainty scores when answering questions when prompted, in addition to retaining the full abilities of the [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct) model. 
 
-### Model Description
+- **Developer:** IBM Research
+- **Model type:** LoRA adapter for [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct)
+- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
-<!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
+### Model Sources
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+- **Paper:** The **Granite Uncertainty 3.0 8b** model is finetuned to provide certainty scores mimicking the output of a calibrator trained via the method in [[Shen et al. ICML 2024] Thermometer: Towards Universal Calibration for Large Language Models](https://arxiv.org/abs/2403.08819)
 
-### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+## Usage
 
-[More Information Needed]
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-### Downstream Use [optional]
+### Intended use
+**Granite Uncertainty 3.0 8b** is lightly tuned so that its behavior closely mimics that of [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
+with the added ability to generate certainty scores for answers to questions when prompted. 
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+**Certainty score definition** The model will respond with a certainty percentage, quantized to 10 possible values (i.e. 5%, 15%, 25%,...95%). 
+This percentage is *calibrated* in the following sense: given a set of answers assigned a certainty score of X%, approximately X% of these answers should be correct. See the eval experiment below for out-of-distribution verification of this behavior.
 
-[More Information Needed]
+**Important note** Certainty is inherently an intrinsic property of a model and its abilitities. **Granite Uncertainty 3.0 8b** is not intended to predict the certainty of responses generated by any other model.
 
-### Out-of-Scope Use
+Answering a question and obtaining a certainty score proceeds as follows. 
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+1. Prompt the model with a system and/or user prompt.
+2. Use the model to generate a response as normal (via the `assistant` role).
+3. Prompt the model to generate a certainty score by generating in the `certainty` role (by appending `<|start_of_role|>certainty<|end_of_role|>` and generating).
+4. The model will respond with a certainty percentage, quantized with steps of 10% (i.e. 5%, 15%, 25%,...95%).
 
-[More Information Needed]
+When not given the certainty generation prompt `<|start_of_role|>certainty<|end_of_role|>`, the model's behavior should mimic that of the base model [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct). 
 
-## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
 
-### Recommendations
+### Quickstart Example
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+The following code describes how to use the Granite Uncertainty model to answer questions and obtain intrinsic calibrated certainty scores. Note that a generic system prompt is included, this is not necessary and can be modified as needed.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+```python
+import torch,os
+from transformers import AutoTokenizer,  AutoModelForCausalLM
+from peft import PeftModel, PeftConfig
 
-## How to Get Started with the Model
+token = os.getenv("HF_MISTRAL_TOKEN")
+BASE_NAME = "ibm-granite/granite-3.0-8b-instruct"
+LORA_NAME = "ibm-granite/granite-uncertainty-3.0-8b-lora"
+device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')
 
-Use the code below to get started with the model.
+# Load model
+token = os.getenv("HF_MISTRAL_TOKEN")
+tokenizer = AutoTokenizer.from_pretrained(BASE_NAME,padding_side='left',trust_remote_code=True, token=token)
+model_base = AutoModelForCausalLM.from_pretrained(BASE_NAME,device_map="auto")
+model_UQ = PeftModel.from_pretrained(model_base, LORA_NAME)
 
-[More Information Needed]
+system_prompt = "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior." #NOTE: this is generic, it can be changed
+question = "What is IBM?"
+print("Question:" + question)
+question_chat = [
+	{
+		"role": "system",
+		"content": system_prompt
+	},
+	{
+		"role": "user",
+		"content": question
+	},
+]
 
-## Training Details
+# Generate answer
+input_text = tokenizer.apply_chat_template(question_chat,tokenize=False,add_generation_prompt=True)
+inputs = tokenizer(input_text, return_tensors="pt")
+output = model_UQ.generate(inputs["input_ids"].to(device), attention_mask=inputs["attention_mask"].to(device), max_new_tokens=80)
+output_text = tokenizer.decode(output[0])
+answer = output_text.split("assistant<|end_of_role|>")[1]
+print("Answer: " + answer)
 
-### Training Data
+# Generate certainty score
+uq_generation_prompt = "<|start_of_role|>certainty<|end_of_role|>"
+uq_chat = [
+    {
+        "role": "system",
+        "content": system_prompt
+    },
+    {
+        "role": "user",
+        "content": question
+    },
+    {
+        "role": "assistant",
+        "content": answer
+    },
+]
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+uq_text = tokenizer.apply_chat_template(uq_chat,tokenize=False) + uq_generation_prompt
+inputs = tokenizer(uq_text, return_tensors="pt")
+output = model_UQ.generate(inputs["input_ids"].to(device), attention_mask=inputs["attention_mask"].to(device), max_new_tokens=1)
+output_text = tokenizer.decode(output[0])
+uq_score = int(output_text[-1])
+print("Certainty: " + str(5 + uq_score * 10) + "%")
+```
 
-[More Information Needed]
 
-### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-#### Preprocessing [optional]
 
-[More Information Needed]
+## Training Details
+The **Granite Uncertainty 3.0 8b** model is a LoRA adapter finetuned to provide certainty scores mimicking the output of a calibrator trained via the method in [[Shen et al. ICML 2024] Thermometer: Towards Universal Calibration for Large Language Models](https://arxiv.org/abs/2403.08819).
 
 
-#### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+### Training Data
+The following datasets were used for calibration and/or finetuning. 
+
+* [BigBench](https://huggingface.co/datasets/tasksource/bigbench)
+* [MRQA](https://huggingface.co/datasets/mrqa-workshop/mrqa)
+* [newsqa](https://huggingface.co/datasets/lucadiliello/newsqa)
+* [trivia_qa](https://huggingface.co/datasets/mandarjoshi/trivia_qa)
+* [search_qa](https://huggingface.co/datasets/lucadiliello/searchqa)
+* [openbookqa](https://huggingface.co/datasets/allenai/openbookqa)
+* [web_questions](https://huggingface.co/datasets/Stanford/web_questions)
+* [smiles-qa](https://huggingface.co/datasets/alxfgh/ChEMBL_Drug_Instruction_Tuning)
+* [orca-math](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k)
+* [ARC-Easy](https://huggingface.co/datasets/allenai/ai2_arc)
+* [commonsense_qa](https://huggingface.co/datasets/tau/commonsense_qa)
+* [social_i_qa](https://huggingface.co/datasets/allenai/social_i_qa)
+* [super_glue](https://huggingface.co/datasets/aps/super_glue)
+* [figqa](https://huggingface.co/datasets/nightingal3/fig-qa)
+* [riddle_sense](https://huggingface.co/datasets/INK-USC/riddle_sense)
+* [ag_news](https://huggingface.co/datasets/fancyzhx/ag_news)
+* [medmcqa](https://huggingface.co/datasets/openlifescienceai/medmcqa)
+* [dream](https://huggingface.co/datasets/dataset-org/dream)
+* [codah](https://huggingface.co/datasets/jaredfern/codah)
+* [piqa](https://huggingface.co/datasets/ybisk/piqa)
 
-#### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-[More Information Needed]
 
 ## Evaluation
 
+The model was evaluated on the [MMLU](https://huggingface.co/datasets/cais/mmlu) datasets (not used in training). Shown are the [Expected Calibration Error (ECE)](https://towardsdatascience.com/expected-calibration-error-ece-a-step-by-step-visual-explanation-with-python-code-c3e9aa12937d) for each task, for the base model (Granite-3.0-8b-instruct) and Granite-Uncertainty-3.0-8b.
+The average ECE across tasks is 0.06 (out of 1). Note that this is smaller than the gap between the quantized certainty outputs (10% quantization steps). 
 <!-- This section describes the evaluation protocols and provides the results. -->
 
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
 
-## Model Card Authors [optional]
 
-[More Information Needed]
 
-## Model Card Contact
+## Model Card Authors 
 
-[More Information Needed]
+Kristjan Greenewald