# Jam 

Jam is a GPT2-like model for research in fine-grained Java analysis. It is intended for fine-grained analysis of Java source code at the level of methods, statements, and variables, as a foundation for downstream tasks like code completion, comment generation, and automated bug repair. 

---
## Jam Training Details

- We trained the jam model using the training procedures from Daniel Grittner's [NanoGPT-LoRA](https://github.com/danielgrittner/nanoGPT-LoRA)

- The dataset used to train our model is our own dataset [jm52m dataset](https://huggingface.co/datasets/apcl/jm52m), which consists of the processed source code of 52 million Java methods. 

- We train the model on [training set](https://huggingface.co/datasets/apcl/jm52m/blob/main/train.bin) for 1 epoch, roughly 300,000 training iterations.
 
- Our [GitHub repo](https://github.com/apcl-research/jam/blob/main) contains the code for re-training using the [raw data](https://huggingface.co/datasets/apcl/jm52m/blob/main/fundats-j1.pkl)

| Hyperparameter | Description | Value |
| ----------- | ----------- |------------|
|e | embedding dimensions              | 1024 |		 
|L | number of layers 			  		| 24 | 		 
|h | attention heads             		| 16 |		 
|c | block size / context length       | 256 |  		 
|b | batch size                        | 4  | 		 
|a | accumulation steps				| 32 |		 
|d | dropout							| 0.20 |		 
|r | learning rate                     | 3e-5 |		 
|y | weight decay						| 1e-1 |	 

We train our models using a single NVidia A5000 GPU. 

---
## Jam Projects

Current projects using the JAM pre-trained model can be found at our Github repository:

https://github.com/apcl-research/jam








{"license": "bigscience-openrail-m", "datasets": ["apcl/jm52m"]}
dataset:apcl/jm52m license:bigscience-openrail-m region:us
TAGS
#dataset-apcl/jm52m #license-bigscience-openrail-m #region-us 

Jam
===


Jam is a GPT2-like model for research in fine-grained Java analysis. It is intended for fine-grained analysis of Java source code at the level of methods, statements, and variables, as a foundation for downstream tasks like code completion, comment generation, and automated bug repair.




---


Jam Training Details
--------------------


* We trained the jam model using the training procedures from Daniel Grittner's NanoGPT-LoRA
* The dataset used to train our model is our own dataset jm52m dataset, which consists of the processed source code of 52 million Java methods.
* We train the model on training set for 1 epoch, roughly 300,000 training iterations.
* Our GitHub repo contains the code for re-training using the raw data


Hyperparameter: e, Description: embedding dimensions, Value: 1024
Hyperparameter: L, Description: number of layers, Value: 24
Hyperparameter: h, Description: attention heads, Value: 16
Hyperparameter: c, Description: block size / context length, Value: 256
Hyperparameter: b, Description: batch size, Value: 4
Hyperparameter: a, Description: accumulation steps, Value: 32
Hyperparameter: d, Description: dropout, Value: 0.20
Hyperparameter: r, Description: learning rate, Value: 3e-5
Hyperparameter: y, Description: weight decay, Value: 1e-1


We train our models using a single NVidia A5000 GPU.




---


Jam Projects
------------


Current projects using the JAM pre-trained model can be found at our Github repository:


URL



TAGS
#dataset-apcl/jm52m #license-bigscience-openrail-m #region-us 

TAGS
#dataset-apcl/jm52m #license-bigscience-openrail-m #region-us 
