kgreenewald
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -39,12 +39,12 @@ with the added ability to generate certainty scores for answers to questions whe
|
|
39 |
**Certainty score definition** The model will respond with a certainty percentage, quantized to 10 possible values (i.e. 5%, 15%, 25%,...95%).
|
40 |
This percentage is *calibrated* in the following sense: given a set of answers assigned a certainty score of X%, approximately X% of these answers should be correct. See the eval experiment below for out-of-distribution verification of this behavior.
|
41 |
|
42 |
-
**Important note** Certainty is inherently an intrinsic property of a model and its abilitities. **Granite Uncertainty 3.0 8b** is not intended to predict the certainty of responses generated by any other
|
43 |
|
44 |
**Usage steps** Answering a question and obtaining a certainty score proceeds as follows.
|
45 |
|
46 |
1. Prompt the model with a system prompt followed by the user prompt. The model is calibrated with the system prompt below.
|
47 |
-
2. Use the model to generate a response as normal (via the `assistant` role).
|
48 |
3. Prompt the model to generate a certainty score by generating in the `certainty` role (by appending `<|start_of_role|>certainty<|end_of_role|>` and generating).
|
49 |
4. The model will respond with a certainty percentage, quantized with steps of 10% (i.e. 5%, 15%, 25%,...95%).
|
50 |
|
|
|
39 |
**Certainty score definition** The model will respond with a certainty percentage, quantized to 10 possible values (i.e. 5%, 15%, 25%,...95%).
|
40 |
This percentage is *calibrated* in the following sense: given a set of answers assigned a certainty score of X%, approximately X% of these answers should be correct. See the eval experiment below for out-of-distribution verification of this behavior.
|
41 |
|
42 |
+
**Important note** Certainty is inherently an intrinsic property of a model and its abilitities. **Granite Uncertainty 3.0 8b** is not intended to predict the certainty of responses generated by any other models besides itself or [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct).
|
43 |
|
44 |
**Usage steps** Answering a question and obtaining a certainty score proceeds as follows.
|
45 |
|
46 |
1. Prompt the model with a system prompt followed by the user prompt. The model is calibrated with the system prompt below.
|
47 |
+
2. Use the model to generate a response as normal (via the `assistant` role), or insert a response from [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct).
|
48 |
3. Prompt the model to generate a certainty score by generating in the `certainty` role (by appending `<|start_of_role|>certainty<|end_of_role|>` and generating).
|
49 |
4. The model will respond with a certainty percentage, quantized with steps of 10% (i.e. 5%, 15%, 25%,...95%).
|
50 |
|