model card draft

README.md

@@ -1,199 +1,109 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- verifcation
+- policy_compliance
+- factchecking
+- instruction_following
+base_model:
+- akjindal53244/Llama-3.1-Storm-8B
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# NAVI verifiers (Nace Automated Verification Intelligence)
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+NAVI (Nace Automated Verification Intelligence) is a solution for policy alignment verification designed to review various types of text against documents and policies, and identify violating content. It is specifically optimized for enterprise applications requiring compliance verification for automated text generation. To push policy verification in the open-source community, we release NAVI-small-preview, an open-weights version of the model we have deployed on the platform. NAVI-small-preview is centered around verifying specifically assitant outputs against some policy documents. The full solution is available on [NAVI platform](naviml.com).
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Developed by:** Nace
+- **Model type:** Policy Alignment Verifier
+- **Language(s) (NLP):** English
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Finetuned from model:** akjindal53244/Llama-3.1-Storm-8B
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
+NAVI-small-preview is used for verifying compliance of assistant outputs with enterprise policy documents. It processes policies and identifies any contradictions, inconsistencies and violations.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Downstream Use
 
-[More Information Needed]
+May be integrated into enterprise-level AI systems to automate compliance checks in diverse industries such as legal, finance, and retail.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+NAVI is not designed for general-purpose factuality verification or tasks unrelated to policy compliance.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+While NAVI excels in policy compliance verification, it may face challenges with unrealistic policy scenarios or contexts outside its training data scope.
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+We recommend to use the model only as a generative classifier outputting the class label, any other outputs are not accounted for during training.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
+The verifier takes the assistant response and document context as inputs to generate a classification label. There are three possible classes: Compliant, Noncompliant, and Irrelevant. For long enterprise documents, we recommend setting up chunk-based vector search for selecting most relevant chunks from the document. Below are the prompts and sample code to launch:
 
-[More Information Needed]
+```
+
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+NAVI was trained on a mix of real-world policy documents and synthetic interactions between a user and an assistant, it includes diverse, realistic, and complex policy examples across multiple industries.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
 #### Preprocessing [optional]
 
-[More Information Needed]
-
+Policies are processed into embeddings for effective decision-making.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+- **Training regime:** [More Information Needed]
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+We have manually collected Policy Alignment Verification dataset consisting of across different use cases for evaluation. We open source the public subset of the dataset. Here we diclose the performance on the public subset, containing 125 examples across six industry-specific scenarios: AT&T, Airbnb, Cadence Bank, Delta Airlines, Verisk, and Walgreens.
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+Evaluation focuses on policy compliance within multi-policy, multi-document contexts.
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+F1 score was used to measure performance, prioritizing detection of noncompliance cases.
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
+NAVI-small-preview achieved an F1 score of 86.8%, outperforming all evaluated alternatives except full-scale NAVI. We evaluate against general-purpose solutions like Claude and Open AI models, as well as some guardrails focusing on factchecking to demonstrate a clear distinction of policy verification from the more common factchecking. Full performance metrics:
 
+| Model                    | F1 Score | Avg Latency (ms) |
+|--------------------------|----------|------------------|
+| NAVI-small-preview      | 86.8     | -                | 
+| NAVI                    | 90.4     | 387.62           |
+| AWS Bedrock Guardrail   | 76.5     | 342.79           |
+| Azure Groundedness      | 71.2     | 232.71           |
+| NeMo (GPT-4o)           | 71.2     | 2669.68          |
+| GPT-4o (few-shot)       | 75.0     | 904.46           |
+| Sonnet 3.5 (few-shot)   | 75.5     | 2926.69          | ()
 
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
+#### Summary
 
-## More Information [optional]
+NAVI demonstrates clear advantages in both performance and efficiency, making it a robust solution for policy compliance verification.
 
-[More Information Needed]
-
-## Model Card Authors [optional]
+## Model Card Contact
 
 [More Information Needed]
 
-## Model Card Contact
-
-[More Information Needed]