PowerInfer

/

SmallThinker-3B-Preview

like 235

Follow

PowerInfer 169

I don't know if the code is correct for this model, the code works but I'm not sure.
https://colab.research.google.com/drive/1EUWwXm71k3KQ8ut3S1Y3Etx_S4Rh235Y?usp=sharing

[{'role': 'system',
  'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant thinking.'},
 {'role': 'user', 'content': 'Give me an example Python LLM'},
 {'role': 'thinking',
  'content': 'As an AI language model developed by Alibaba Cloud, I\'m here to help you with your questions and provide assistance with various tasks related to programming, including generating examples in Python.\n\nTo answer this question directly: Here\'s an example of a simple Python program that prints "Hello, World!" to the console:\n\n```python\nprint("Hello, World!")\n```\n\nThis is a basic example to get started with writing Python code. The `print` function is used to output text to the console.\n\nIf you have any specific requirements or need further assistance with Python programming, feel free to ask!'},
 {'role': 'assistant',
  'content': 'Sure, I\'d be happy to help with that! So, you\'re looking for an example of a Python language model, right? Well, there are several options available, each with its own strengths and use cases. Let\'s explore a few popular ones.\n\nFirst off, there\'s **Hugging Face Transformers**, which is a fantastic library for working with pre-trained models in NLP tasks like translation, summarization, and more. It supports a wide range of models from various manufacturers, including BERT, GPT, RoBERTa, and many others.\n\nTo get started with Hugging Face Transformers, you\'ll need to install the library using pip:\n\n```bash\npip install transformers\n```\n\nOnce installed, you can load a pre-trained model and tokenizer like this:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n# Load pre-trained model and tokenizer\nmodel_name = "distilbert-base-uncased-finetuned-sst-2-english"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\nmodel = AutoModelForSequenceClassification.from_pretrained(model_name)\n\n# Example usage\ninputs = tokenizer("I love this product!", return_tensors="pt")\noutputs = model(**inputs)\n```\n\nThat\'s pretty straightforward, isn\'t it?\n\nAnother powerful tool is **TensorFlow Text**, part of the TensorFlow ecosystem. TensorFlow has been around for a while and offers extensive support for machine learning tasks, including natural language processing.\n\nTo install TensorFlow Text, you would do something like this:\n\n```bash\npip install tensorflow-text\n```\n\nWith TensorFlow Text, you can perform tasks such as tokenization, text classification, and even sentiment analysis. Here\'s a simple example:\n\n```python\nimport tensorflow_text as tf_text\nfrom tensorflow.keras.layers import TextVectorization\n\n# Initialize the vectorizer\nmax_document_length = 100\nvocab_size = 10000\n\nvectorizer = TextVectorization(max_tokens=vocab_size, output_sequence_length=max_document_length)\n\n# Example data\ntexts = ["This is a sample sentence.", "Another one for testing."]\n\n# Fit the vectorizer on your data\nvectorizer.adapt(texts)\n\n# Transform the texts into sequences\nsequences = vectorizer(texts)\n\nprint(sequences)\n```\n\nLastly, if you prefer using PyTorch, there\'s also the **Transformers Library** built specifically for PyTorch, which provides state-of-the-art models for various NLP tasks.\n\nInstalling Transformers via PyPI is easy:\n\n```bash\npip install transformers\n```\n\nHere\'s how you might use it in PyTorch:\n\n```python\nfrom transformers import BertTokenizer, BertForTokenClassification\n\n# Load pre-trained model and tokenizer\ntokenizer = BertTokenizer.from_pretrained(\'bert-base-uncased\')\nmodel = BertForTokenClassification.from_pretrained(\'bert-base-uncased\')\n\n# Example usage\ntokens = tokenizer.tokenize("I love this product!")\ninput_ids = tokenizer.convert_tokens_to_ids(tokens)\nlabels = [model.config.label_map[token] for token in tokens]\n\n# Prepare the input tensor\ninput_tensor = torch.tensor([input_ids])\nlabels_tensor = torch.tensor(labels)\n\n# Forward pass\noutputs = model(input_tensor, labels=labels_tensor)\nloss = outputs.loss\npredictions = outputs.logits.argmax(dim=-1)\n```\n\nEach of these libraries has its own way of handling models and tasks, so the choice depends on your specific needs and existing setup. If you have any particular requirements or constraints, feel free to let me know, and I can provide more tailored advice!\n\n**Summary:**\n\n- **Hugging Face Transformers:** Comprehensive, widely-used library for NLP tasks.\n- **TensorFlow Text:** Part of TensorFlow, good for integrating with other TensorFlow tools.\n- **PyTorch Transformers:** Built specifically for PyTorch, offering seamless integration.\n\nChoose based on your project\'s requirements and your team\'s familiarity with the libraries. Happy coding!'}]

We plan to open a space demo on huggingface let everyone try the model.

You can try on: https://huggingface.co/spaces/PowerInfer/SmallThinker-Demo

In the code they have "system", "user", "assistant" don't they use the other special tokens like "thinking"?

Was the model trained normally with only the tokens "user", "system" and "assistant"?

Yes, in our early experiments we tried putting Chain of Thought in the "Thinking" section, but the experiments were not good enough for reasons that remain unclear. Currently, we follow the QWQ output format, training the model to provide responses directly in the "assistant" section.

Thank you very much for the response.

I would like to know if you plan to use a 1-5 scoring system for the prompts where only the best are selected to make fine tune?

Do you plan to use special tokens with thinking in other versions?

I also notice that they have modified the chat_Template and it is not "thinking" for it to appear.

Thank you for providing such excellent insights. Yes, we will continue to iterate to develop better models. :)