# Model Card for yolochess_mlm_azure-cloud-35

<!-- Provide a quick summary of what the model is/does. -->

This model with 66M parameters is pre-trained from scratch with Masked Language Modeling on Chess Positions in [FEN](https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation) format.  
It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.

# Model Details

## Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** Jonathan Rahn
- **Model type:** Distilbert
- **Language(s) (NLP):** Chess [FEN](https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation)
- **License:** MIT

# Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

## Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

This model is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  

## Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.

## Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Anything other than Chess Positions in standard [FEN](https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation) format.

# Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

n/a

## Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

n/a

## How to Get Started with the Model

Use the code below to get started with the model.

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("jrahn/yolochess_mlm_azure-cloud-35")
model = AutoModelForMaskedLM.from_pretrained("jrahn/yolochess_mlm_azure-cloud-35")
```

```python
from transformers import pipeline
pipe = pipeline("fill-mask", "jrahn/yolochess_mlm_azure-cloud-35")
pipe("6k1/8/8/1pB3[MASK]P/1P3P2/8/8/8 w - - 1 74")
```

# Training Details

## Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[Lichess-Elite 22-11 Dataset](https://huggingface.co/datasets/jrahn/yolochess_lichess-elite_2211)

## Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Masked Language Modeling objective with 15% masked token ratio.

### Preprocessing

Tokenize `data["train"]["fen"]` with max-length padding to 200 tokens with default `distilbert-base-cased` tokenizer. 
Inefficient: Most of the vocab is never observed in FEN, wasting embedding parameters. 
The sequence length / pos embedding size of model and sequence length of data preprocessing leads to lots of padding and wasted parameters. FENs should be shorter than 90 characters.
Experiments with reduced max-length in tokenization show performance gains.

### Speeds, Sizes, Times

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

Training for 172500 steps at batch-size 128 (22M examples, 1 epoch) took ~10 hrs on 1x RTX 4090, using 20GB VRAM, with final MLM-loss: 0.2567.

# Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** 1x RTX 4090
- **Hours used:** 10
- **Cloud Provider:** local
- **Compute Region:** local
- **Carbon Emitted:** 1.5kg

# Technical Specifications

## Model Architecture and Objective

Distilbert, Masked Language Modeling

{"license": "mit", "library_name": "transformers", "tags": ["chess"], "datasets": ["jrahn/yolochess_lichess-elite_2211"], "widget": [{"text": "rnbqkbnr/pppppppp/8/8/8/[MASK]/PPPPPPPP/RNBQKBNR w KQkq - 0 1", "example_title": "MLM: Masked = 8"}, {"text": "6k1/8/8/1pB3[MASK]P/1P3P2/8/8/8 w - - 1 74", "example_title": "MLM: Masked = K"}]}
transformers pytorch safetensors distilbert fill-mask chess dataset:jrahn/yolochess_lichess-elite_2211 arxiv:1910.09700 license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #chess #dataset-jrahn/yolochess_lichess-elite_2211 #arxiv-1910.09700 #license-mit #autotrain_compatible #endpoints_compatible #region-us 

# Model Card for yolochess_mlm_azure-cloud-35



This model with 66M parameters is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  
It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.

# Model Details

## Model Description





- Developed by: Jonathan Rahn
- Model type: Distilbert
- Language(s) (NLP): Chess FEN
- License: MIT

# Uses



## Direct Use



This model is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  

## Downstream Use



It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.

## Out-of-Scope Use



Anything other than Chess Positions in standard FEN format.

# Bias, Risks, and Limitations



n/a

## Recommendations



n/a

## How to Get Started with the Model

Use the code below to get started with the model.





# Training Details

## Training Data



Lichess-Elite 22-11 Dataset

## Training Procedure



Masked Language Modeling objective with 15% masked token ratio.

### Preprocessing

Tokenize 'data["train"]["fen"]' with max-length padding to 200 tokens with default 'distilbert-base-cased' tokenizer. 
Inefficient: Most of the vocab is never observed in FEN, wasting embedding parameters. 
The sequence length / pos embedding size of model and sequence length of data preprocessing leads to lots of padding and wasted parameters. FENs should be shorter than 90 characters.
Experiments with reduced max-length in tokenization show performance gains.

### Speeds, Sizes, Times



Training for 172500 steps at batch-size 128 (22M examples, 1 epoch) took ~10 hrs on 1x RTX 4090, using 20GB VRAM, with final MLM-loss: 0.2567.

# Environmental Impact



Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

- Hardware Type: 1x RTX 4090
- Hours used: 10
- Cloud Provider: local
- Compute Region: local
- Carbon Emitted: 1.5kg

# Technical Specifications

## Model Architecture and Objective

Distilbert, Masked Language Modeling

# Model Card for yolochess_mlm_azure-cloud-35



This model with 66M parameters is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  
It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves. # Model Details ## Model Description





- Developed by: Jonathan Rahn
- Model type: Distilbert
- Language(s) (NLP): Chess FEN
- License: MIT # Uses ## Direct Use



This model is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format. ## Downstream Use



It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves. ## Out-of-Scope Use



Anything other than Chess Positions in standard FEN format. # Bias, Risks, and Limitations



n/a ## Recommendations



n/a ## How to Get Started with the Model

Use the code below to get started with the model. # Training Details ## Training Data



Lichess-Elite 22-11 Dataset ## Training Procedure



Masked Language Modeling objective with 15% masked token ratio. ### Preprocessing

Tokenize 'data["train"]["fen"]' with max-length padding to 200 tokens with default 'distilbert-base-cased' tokenizer. 
Inefficient: Most of the vocab is never observed in FEN, wasting embedding parameters. 
The sequence length / pos embedding size of model and sequence length of data preprocessing leads to lots of padding and wasted parameters. FENs should be shorter than 90 characters.
Experiments with reduced max-length in tokenization show performance gains. ### Speeds, Sizes, Times



Training for 172500 steps at batch-size 128 (22M examples, 1 epoch) took ~10 hrs on 1x RTX 4090, using 20GB VRAM, with final MLM-loss: 0.2567. # Environmental Impact



Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

- Hardware Type: 1x RTX 4090
- Hours used: 10
- Cloud Provider: local
- Compute Region: local
- Carbon Emitted: 1.5kg # Technical Specifications ## Model Architecture and Objective

Distilbert, Masked Language Modeling
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #chess #dataset-jrahn/yolochess_lichess-elite_2211 #arxiv-1910.09700 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 # Model Card for yolochess_mlm_azure-cloud-35



This model with 66M parameters is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  
It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves. # Model Details ## Model Description





- Developed by: Jonathan Rahn
- Model type: Distilbert
- Language(s) (NLP): Chess FEN
- License: MIT # Uses ## Direct Use



This model is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format. ## Downstream Use



It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves. ## Out-of-Scope Use



Anything other than Chess Positions in standard FEN format. # Bias, Risks, and Limitations



n/a ## Recommendations



n/a ## How to Get Started with the Model

Use the code below to get started with the model. # Training Details ## Training Data



Lichess-Elite 22-11 Dataset ## Training Procedure



Masked Language Modeling objective with 15% masked token ratio. ### Preprocessing

Tokenize 'data["train"]["fen"]' with max-length padding to 200 tokens with default 'distilbert-base-cased' tokenizer. 
Inefficient: Most of the vocab is never observed in FEN, wasting embedding parameters. 
The sequence length / pos embedding size of model and sequence length of data preprocessing leads to lots of padding and wasted parameters. FENs should be shorter than 90 characters.
Experiments with reduced max-length in tokenization show performance gains. ### Speeds, Sizes, Times



Training for 172500 steps at batch-size 128 (22M examples, 1 epoch) took ~10 hrs on 1x RTX 4090, using 20GB VRAM, with final MLM-loss: 0.2567. # Environmental Impact



Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

- Hardware Type: 1x RTX 4090
- Hours used: 10
- Cloud Provider: local
- Compute Region: local
- Carbon Emitted: 1.5kg # Technical Specifications ## Model Architecture and Objective

Distilbert, Masked Language Modeling
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #chess #dataset-jrahn/yolochess_lichess-elite_2211 #arxiv-1910.09700 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
# Model Card for yolochess_mlm_azure-cloud-35



This model with 66M parameters is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.  
It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.# Model Details## Model Description





- Developed by: Jonathan Rahn
- Model type: Distilbert
- Language(s) (NLP): Chess FEN
- License: MIT# Uses## Direct Use



This model is pre-trained from scratch with Masked Language Modeling on Chess Positions in FEN format.## Downstream Use



It is supposed to be used for downstream fine-tuning, e.g. Text Classification for human moves.## Out-of-Scope Use



Anything other than Chess Positions in standard FEN format.# Bias, Risks, and Limitations



n/a## Recommendations



n/a## How to Get Started with the Model

Use the code below to get started with the model.# Training Details## Training Data



Lichess-Elite 22-11 Dataset## Training Procedure



Masked Language Modeling objective with 15% masked token ratio.### Preprocessing

Tokenize 'data["train"]["fen"]' with max-length padding to 200 tokens with default 'distilbert-base-cased' tokenizer. 
Inefficient: Most of the vocab is never observed in FEN, wasting embedding parameters. 
The sequence length / pos embedding size of model and sequence length of data preprocessing leads to lots of padding and wasted parameters. FENs should be shorter than 90 characters.
Experiments with reduced max-length in tokenization show performance gains.### Speeds, Sizes, Times



Training for 172500 steps at batch-size 128 (22M examples, 1 epoch) took ~10 hrs on 1x RTX 4090, using 20GB VRAM, with final MLM-loss: 0.2567.# Environmental Impact



Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

- Hardware Type: 1x RTX 4090
- Hours used: 10
- Cloud Provider: local
- Compute Region: local
- Carbon Emitted: 1.5kg# Technical Specifications## Model Architecture and Objective

Distilbert, Masked Language Modeling