# Dataset Card for KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos

## Dataset Description

- **Homepage:** https://dirtmaxim.github.io/kabr/
- **Repository:** https://github.com/dirtmaxim/kabr-tools
- **Paper:** https://openaccess.thecvf.com/content/WACV2024W/CV4Smalls/papers/Kholiavchenko_KABR_In-Situ_Dataset_for_Kenyan_Animal_Behavior_Recognition_From_Drone_WACVW_2024_paper.pdf
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary

We present a novel high-quality dataset for animal behavior recognition from drone videos. 
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras. 
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances. 
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process. 
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations. 
The dataset was collected using drones that flew over the animals in the [Mpala Research Centre](https://mpala.org/) in Kenya, providing high-quality video footage of the animal's natural behaviors.
The drone footage is captured at a resolution of 5472 x 3078 pixels, and the videos were recorded at a frame rate of 29.97 frames per second.


<!--This dataset card aims to be a base template for new datasets. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1).-->

### Supported Tasks and Leaderboards

The results of our evaluation using I3D, SlowFast, and X3D architectures are given in the table below. For each one, the model was trained for 120 epochs with batch size of 5. For more information on these results, see our [paper](coming soon).

| Method | All | Giraffes | Plains Zebras | Grevy’s Zebras |
| ---- | ---- | ---- | ---- | ---- |
| I3D (16x5)  |  53.41 | 61.82 | 58.75 | 46.73 |
| SlowFast (16x5, 4x5) | 52.92 | 61.15 | 60.60 | 47.42 |
| X3D (16x5) | 61.9 | 65.1 | 63.11 | 51.16 |

### Languages

English

## Dataset Structure

Under `KABR/dataset/image/`, the data has been archived into `.zip` files, which are split into 2GB files. These must be recombined and extracted. 
After cloning and navigating into the repository, you can use the following commands to do the reconstruction:
```bash
cd KABR/dataset/image/
cat giraffes_part_* > giraffes.zip
md5sum giraffes.zip # Compare this to what's shown with `cat giraffes_md5.txt`
unzip giraffes.zip
rm -rf giraffes_part_*

# Similarly for `zebras_grevys_part_*` and `zebras_plains_part_*`
```

Alternatively, there is a download script, `download.py`, which allows a download of the entire dataset in its established format without requiring one to clone the repository (cloning requires _at least_ double the size of the dataset to store). To proceed with this approach, download `download.py` to the system where you want to access the data.
Then, in the same directory as the script, run the following to begin the download:
```
pip install requests
python download.py
```

This script then downloads all the files present in the repository (without making a clone of the `.git` directory, etc.), concatenates the part files to their ZIP archives, verifies the MD5 checksums, extracts, and cleans up so that the folder structure, as described below, is present.

Note that it will require approximately 116GB of free space to complete this process, though the final dataset will only take about 61GB of disk space (the script removes the extra files after checking the download was successful).

The KABR dataset follows the Charades format:

```
KABR
    /dataset
        /image
            /video_1
                /image_1.jpg
                /image_2.jpg
                ...
                /image_n.jpg
            /video_2
                /image_1.jpg
                /image_2.jpg
                ...
                /image_n.jpg
            ...
            /video_n
                /image_1.jpg
                /image_2.jpg
                /image_3.jpg
                ...
                /image_n.jpg
    /annotation
        /classes.json
        /train.csv
        /val.csv
```

The dataset can be directly loaded and processed by the [SlowFast](https://github.com/facebookresearch/SlowFast) framework.

**Informational Files**
* `KABR/configs`: examples of SlowFast framework configs.
* `KABR/annotation/distribution.xlsx`: distribution of classes for all videos.

**Scripts:**
* `image2video.py`: Encode image sequences into the original video.
  * For example, `[image/G0067.1, image/G0067.2, ..., image/G0067.24]` will be encoded into `video/G0067.mp4`.
* `image2visual.py`: Encode image sequences into the original video with corresponding annotations.
  * For example, `[image/G0067.1, image/G0067.2, ..., image/G0067.24]` will be encoded into `visual/G0067.mp4`.

### Data Instances

**Naming:** Within the image folder, the `video_n` folders are named as follows (X indicates a number):
* G0XXX.X - Giraffes
* ZP0XXX.X - Plains Zebras
* ZG0XXX.X - Grevy's Zebras
* Within each of these folders the images are simply `X.jpg`.

**Note:** The dataset consists of a total of 1,139,893 frames captured from drone videos. There are 488,638 frames of Grevy's zebras, 492,507 frames of plains zebras, and 158,748 frames of giraffes.


### Data Fields

There are 14,764 unique behavioral sequences in the dataset. These consist of eight distinct behaviors:
- Walk
- Trot
- Run: animal is moving at a cantor or gallop
- Graze: animal is eating grass or other vegetation
- Browse: animal is eating trees or bushes
- Head Up: animal is looking around or observe surroundings
- Auto-Groom: animal is grooming itself (licking, scratching, or rubbing)
- Occluded: animal is not fully visible

### Data Splits

Training and validation sets are indicated by their respective CSV files (`train.csv` and `val.csv`), located within the `annotation` folder.

## Dataset Creation

### Curation Rationale

We present a novel high-quality dataset for animal behavior recognition from drone videos. 
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras. 
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances. 
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process. 
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations. 
The dataset was collected using drones that flew over the animals in the [Mpala Research Centre](https://mpala.org/) in Kenya, providing high-quality video footage of the animal's natural behaviors. 
We believe that this dataset will be a valuable resource for researchers working on animal behavior recognition, as it provides a diverse and high-quality set of annotated videos that can be used for evaluating deep learning models. 
Additionally, the dataset can be used to study the behavior patterns of Kenyan animals and can help to inform conservation efforts and wildlife management strategies. 

<!-- [To be added:] -->

We provide a detailed description of the dataset and its annotation process, along with some initial experiments on the dataset using conventional deep learning models. 
The results demonstrate the effectiveness of the dataset for animal behavior recognition and highlight the potential for further research in this area.

### Source Data

#### Initial Data Collection and Normalization

Data was collected from 6 January 2023 through 21 January 2023 at the [Mpala Research Centre](https://mpala.org/) in Kenya under a Nacosti research license. We used DJI Mavic 2S drones equipped with cameras to record 5.4K resolution videos (5472 x 3078 pixels) from varying altitudes and distances of 10 to 50 meters from the animals (distance was determined by circumstances and safety regulations).

Mini-scenes were extracted from these videos to reduce the impact of drone movement and facilitate human annotation. Animals were detected in frame using YOLOv8, then the SORT tracking algorithm was applied to follow their movement. A 400 by 300 pixel window, centered on the animal, was then extracted; this is the mini-scene.

<!--
#### Who are the source language producers?

[More Information Needed]
-->

### Annotations

#### Annotation process

In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.

<!--
#### Who are the annotators?

[More Information Needed]
-->

### Personal and Sensitive Information

Though there are endangered species included in this data, exact locations are not provided and their safety is assured by their location within the preserve.

## Considerations for Using the Data
<!--
### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]
-->

### Other Known Limitations

This data exhibits a long-tailed distribution due to the natural variation in frequency of the observed behaviors.

## Additional Information

### Authors

* Maksim Kholiavchenko (Rensselaer Polytechnic Institute) - ORCID: 0000-0001-6757-1957
* Jenna Kline (The Ohio State University)
* Michelle Ramirez (The Ohio State University)
* Sam Stevens (The Ohio State University)
* Alec Sheets (The Ohio State University) - ORCID: 0000-0002-3737-1484
* Reshma Ramesh Babu (The Ohio State University) - ORCID: 0000-0002-2517-5347
* Namrata Banerji (The Ohio State University) - ORCID: 0000-0001-6813-0010
* Elizabeth Campolongo (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0846-2413
* Matthew Thompson (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0583-8585
* Nina Van Tiel (Eidgenössische Technische Hochschule Zürich) - ORCID: 0000-0001-6393-5629
* Jackson Miliko (Mpala Research Centre)
* Eduardo Bessa (Universidade de Brasília) - ORCID: 0000-0003-0606-5860
* Tanya Berger-Wolf (The Ohio State University) - ORCID: 0000-0001-7610-1412
* Daniel Rubenstein (Princeton University) - ORCID: 0000-0001-9049-5219
* Charles Stewart (Rensselaer Polytechnic Institute)

### Licensing Information

This dataset is dedicated to the public domain for the benefit of scientific pursuits. We ask that you cite the dataset and journal paper using the below citations if you make use of it in your research.

### Citation Information

#### Dataset
```
@misc{KABR_Data,
  author = {Kholiavchenko, Maksim and Kline, Jenna and Ramirez, Michelle and Stevens, Sam and Sheets, Alec and Babu, Reshma and Banerji, Namrata and Campolongo, Elizabeth and Thompson, Matthew and Van Tiel, Nina and Miliko, Jackson and Bessa, Eduardo and Duporge, Isla and Berger-Wolf, Tanya and Rubenstein, Daniel and Stewart, Charles},
  title = {KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos},
  year = {2023},
  url = {https://huggingface.co/datasets/imageomics/KABR},
  doi = {10.57967/hf/1010},
  publisher = {Hugging Face}
}
```

#### Paper
```
@inproceedings{kholiavchenko2024kabr,
  title={KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos},
  author={Kholiavchenko, Maksim and Kline, Jenna and Ramirez, Michelle and Stevens, Sam and Sheets, Alec and Babu, Reshma and Banerji, Namrata and Campolongo, Elizabeth and Thompson, Matthew and Van Tiel, Nina and Miliko, Jackson and Bessa, Eduardo and Duporge, Isla and Berger-Wolf, Tanya and Rubenstein, Daniel and Stewart, Charles},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={31-40},
  year={2024}
}
```

### Contributions

The [Imageomics Institute](https://imageomics.org) is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

task_categories:video-classification size_categories:1M<n<10M license:cc0-1.0 zebra giraffe plains zebra Grevy's zebra video animal behavior behavior recognition annotation annotated video conservation drone UAV imbalanced Kenya Mpala Research Centre doi:10.57967/hf/1010 region:us
{"license": "cc0-1.0", "size_categories": ["1M<n<10M"], "task_categories": ["video-classification"], "pretty_name": "KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos", "tags": ["zebra", "giraffe", "plains zebra", "Grevy's zebra", "video", "animal behavior", "behavior recognition", "annotation", "annotated video", "conservation", "drone", "UAV", "imbalanced", "Kenya", "Mpala Research Centre"]}
TAGS
#task_categories-video-classification #size_categories-1M<n<10M #license-cc0-1.0 #zebra #giraffe #plains zebra #Grevy's zebra #video #animal behavior #behavior recognition #annotation #annotated video #conservation #drone #UAV #imbalanced #Kenya #Mpala Research Centre #doi-10.57967/hf/1010 #region-us 

Dataset Card for KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition from Drone Videos
===============================================================================================


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard:
* Point of Contact:


### Dataset Summary


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
The drone footage is captured at a resolution of 5472 x 3078 pixels, and the videos were recorded at a frame rate of 29.97 frames per second.


### Supported Tasks and Leaderboards


The results of our evaluation using I3D, SlowFast, and X3D architectures are given in the table below. For each one, the model was trained for 120 epochs with batch size of 5. For more information on these results, see our paper.



### Languages


English


Dataset Structure
-----------------


Under 'KABR/dataset/image/', the data has been archived into '.zip' files, which are split into 2GB files. These must be recombined and extracted.
After cloning and navigating into the repository, you can use the following commands to do the reconstruction:


Alternatively, there is a download script, 'URL', which allows a download of the entire dataset in its established format without requiring one to clone the repository (cloning requires *at least* double the size of the dataset to store). To proceed with this approach, download 'URL' to the system where you want to access the data.
Then, in the same directory as the script, run the following to begin the download:


This script then downloads all the files present in the repository (without making a clone of the '.git' directory, etc.), concatenates the part files to their ZIP archives, verifies the MD5 checksums, extracts, and cleans up so that the folder structure, as described below, is present.


Note that it will require approximately 116GB of free space to complete this process, though the final dataset will only take about 61GB of disk space (the script removes the extra files after checking the download was successful).


The KABR dataset follows the Charades format:


The dataset can be directly loaded and processed by the SlowFast framework.


Informational Files


* 'KABR/configs': examples of SlowFast framework configs.
* 'KABR/annotation/URL': distribution of classes for all videos.


Scripts:


* 'URL': Encode image sequences into the original video.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'video/G0067.mp4'.
* 'URL': Encode image sequences into the original video with corresponding annotations.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'visual/G0067.mp4'.


### Data Instances


Naming: Within the image folder, the 'video\_n' folders are named as follows (X indicates a number):


* G0XXX.X - Giraffes
* ZP0XXX.X - Plains Zebras
* ZG0XXX.X - Grevy's Zebras
* Within each of these folders the images are simply 'X.jpg'.


Note: The dataset consists of a total of 1,139,893 frames captured from drone videos. There are 488,638 frames of Grevy's zebras, 492,507 frames of plains zebras, and 158,748 frames of giraffes.


### Data Fields


There are 14,764 unique behavioral sequences in the dataset. These consist of eight distinct behaviors:


* Walk
* Trot
* Run: animal is moving at a cantor or gallop
* Graze: animal is eating grass or other vegetation
* Browse: animal is eating trees or bushes
* Head Up: animal is looking around or observe surroundings
* Auto-Groom: animal is grooming itself (licking, scratching, or rubbing)
* Occluded: animal is not fully visible


### Data Splits


Training and validation sets are indicated by their respective CSV files ('URL' and 'URL'), located within the 'annotation' folder.


Dataset Creation
----------------


### Curation Rationale


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
We believe that this dataset will be a valuable resource for researchers working on animal behavior recognition, as it provides a diverse and high-quality set of annotated videos that can be used for evaluating deep learning models.
Additionally, the dataset can be used to study the behavior patterns of Kenyan animals and can help to inform conservation efforts and wildlife management strategies.


We provide a detailed description of the dataset and its annotation process, along with some initial experiments on the dataset using conventional deep learning models.
The results demonstrate the effectiveness of the dataset for animal behavior recognition and highlight the potential for further research in this area.


### Source Data


#### Initial Data Collection and Normalization


Data was collected from 6 January 2023 through 21 January 2023 at the Mpala Research Centre in Kenya under a Nacosti research license. We used DJI Mavic 2S drones equipped with cameras to record 5.4K resolution videos (5472 x 3078 pixels) from varying altitudes and distances of 10 to 50 meters from the animals (distance was determined by circumstances and safety regulations).


Mini-scenes were extracted from these videos to reduce the impact of drone movement and facilitate human annotation. Animals were detected in frame using YOLOv8, then the SORT tracking algorithm was applied to follow their movement. A 400 by 300 pixel window, centered on the animal, was then extracted; this is the mini-scene.


### Annotations


#### Annotation process


In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.


### Personal and Sensitive Information


Though there are endangered species included in this data, exact locations are not provided and their safety is assured by their location within the preserve.


Considerations for Using the Data
---------------------------------


### Other Known Limitations


This data exhibits a long-tailed distribution due to the natural variation in frequency of the observed behaviors.


Additional Information
----------------------


### Authors


* Maksim Kholiavchenko (Rensselaer Polytechnic Institute) - ORCID: 0000-0001-6757-1957
* Jenna Kline (The Ohio State University)
* Michelle Ramirez (The Ohio State University)
* Sam Stevens (The Ohio State University)
* Alec Sheets (The Ohio State University) - ORCID: 0000-0002-3737-1484
* Reshma Ramesh Babu (The Ohio State University) - ORCID: 0000-0002-2517-5347
* Namrata Banerji (The Ohio State University) - ORCID: 0000-0001-6813-0010
* Elizabeth Campolongo (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0846-2413
* Matthew Thompson (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0583-8585
* Nina Van Tiel (Eidgenössische Technische Hochschule Zürich) - ORCID: 0000-0001-6393-5629
* Jackson Miliko (Mpala Research Centre)
* Eduardo Bessa (Universidade de Brasília) - ORCID: 0000-0003-0606-5860
* Tanya Berger-Wolf (The Ohio State University) - ORCID: 0000-0001-7610-1412
* Daniel Rubenstein (Princeton University) - ORCID: 0000-0001-9049-5219
* Charles Stewart (Rensselaer Polytechnic Institute)


### Licensing Information


This dataset is dedicated to the public domain for the benefit of scientific pursuits. We ask that you cite the dataset and journal paper using the below citations if you make use of it in your research.


#### Dataset


#### Paper


### Contributions


The Imageomics Institute is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.



### Dataset Summary


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
The drone footage is captured at a resolution of 5472 x 3078 pixels, and the videos were recorded at a frame rate of 29.97 frames per second. ### Supported Tasks and Leaderboards


The results of our evaluation using I3D, SlowFast, and X3D architectures are given in the table below. For each one, the model was trained for 120 epochs with batch size of 5. For more information on these results, see our paper. ### Languages


English


Dataset Structure
-----------------


Under 'KABR/dataset/image/', the data has been archived into '.zip' files, which are split into 2GB files. These must be recombined and extracted.
After cloning and navigating into the repository, you can use the following commands to do the reconstruction:


Alternatively, there is a download script, 'URL', which allows a download of the entire dataset in its established format without requiring one to clone the repository (cloning requires *at least* double the size of the dataset to store). To proceed with this approach, download 'URL' to the system where you want to access the data.
Then, in the same directory as the script, run the following to begin the download:


This script then downloads all the files present in the repository (without making a clone of the '.git' directory, etc.), concatenates the part files to their ZIP archives, verifies the MD5 checksums, extracts, and cleans up so that the folder structure, as described below, is present.


Note that it will require approximately 116GB of free space to complete this process, though the final dataset will only take about 61GB of disk space (the script removes the extra files after checking the download was successful).


The KABR dataset follows the Charades format:


The dataset can be directly loaded and processed by the SlowFast framework.


Informational Files


* 'KABR/configs': examples of SlowFast framework configs.
* 'KABR/annotation/URL': distribution of classes for all videos.


Scripts:


* 'URL': Encode image sequences into the original video.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'video/G0067.mp4'.
* 'URL': Encode image sequences into the original video with corresponding annotations.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'visual/G0067.mp4'. ### Data Instances


Naming: Within the image folder, the 'video\_n' folders are named as follows (X indicates a number):


* G0XXX.X - Giraffes
* ZP0XXX.X - Plains Zebras
* ZG0XXX.X - Grevy's Zebras
* Within each of these folders the images are simply 'X.jpg'.


Note: The dataset consists of a total of 1,139,893 frames captured from drone videos. There are 488,638 frames of Grevy's zebras, 492,507 frames of plains zebras, and 158,748 frames of giraffes. ### Data Fields


There are 14,764 unique behavioral sequences in the dataset. These consist of eight distinct behaviors:


* Walk
* Trot
* Run: animal is moving at a cantor or gallop
* Graze: animal is eating grass or other vegetation
* Browse: animal is eating trees or bushes
* Head Up: animal is looking around or observe surroundings
* Auto-Groom: animal is grooming itself (licking, scratching, or rubbing)
* Occluded: animal is not fully visible ### Data Splits


Training and validation sets are indicated by their respective CSV files ('URL' and 'URL'), located within the 'annotation' folder.


Dataset Creation
---------------- ### Curation Rationale


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
We believe that this dataset will be a valuable resource for researchers working on animal behavior recognition, as it provides a diverse and high-quality set of annotated videos that can be used for evaluating deep learning models.
Additionally, the dataset can be used to study the behavior patterns of Kenyan animals and can help to inform conservation efforts and wildlife management strategies.


We provide a detailed description of the dataset and its annotation process, along with some initial experiments on the dataset using conventional deep learning models.
The results demonstrate the effectiveness of the dataset for animal behavior recognition and highlight the potential for further research in this area. ### Source Data #### Initial Data Collection and Normalization


Data was collected from 6 January 2023 through 21 January 2023 at the Mpala Research Centre in Kenya under a Nacosti research license. We used DJI Mavic 2S drones equipped with cameras to record 5.4K resolution videos (5472 x 3078 pixels) from varying altitudes and distances of 10 to 50 meters from the animals (distance was determined by circumstances and safety regulations).


Mini-scenes were extracted from these videos to reduce the impact of drone movement and facilitate human annotation. Animals were detected in frame using YOLOv8, then the SORT tracking algorithm was applied to follow their movement. A 400 by 300 pixel window, centered on the animal, was then extracted; this is the mini-scene. ### Annotations #### Annotation process


In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations. ### Personal and Sensitive Information


Though there are endangered species included in this data, exact locations are not provided and their safety is assured by their location within the preserve.


Considerations for Using the Data
--------------------------------- ### Other Known Limitations


This data exhibits a long-tailed distribution due to the natural variation in frequency of the observed behaviors.


Additional Information
---------------------- ### Authors


* Maksim Kholiavchenko (Rensselaer Polytechnic Institute) - ORCID: 0000-0001-6757-1957
* Jenna Kline (The Ohio State University)
* Michelle Ramirez (The Ohio State University)
* Sam Stevens (The Ohio State University)
* Alec Sheets (The Ohio State University) - ORCID: 0000-0002-3737-1484
* Reshma Ramesh Babu (The Ohio State University) - ORCID: 0000-0002-2517-5347
* Namrata Banerji (The Ohio State University) - ORCID: 0000-0001-6813-0010
* Elizabeth Campolongo (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0846-2413
* Matthew Thompson (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0583-8585
* Nina Van Tiel (Eidgenössische Technische Hochschule Zürich) - ORCID: 0000-0001-6393-5629
* Jackson Miliko (Mpala Research Centre)
* Eduardo Bessa (Universidade de Brasília) - ORCID: 0000-0003-0606-5860
* Tanya Berger-Wolf (The Ohio State University) - ORCID: 0000-0001-7610-1412
* Daniel Rubenstein (Princeton University) - ORCID: 0000-0001-9049-5219
* Charles Stewart (Rensselaer Polytechnic Institute) ### Licensing Information


This dataset is dedicated to the public domain for the benefit of scientific pursuits. We ask that you cite the dataset and journal paper using the below citations if you make use of it in your research. #### Dataset #### Paper ### Contributions


The Imageomics Institute is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
TAGS
#task_categories-video-classification #size_categories-1M<n<10M #license-cc0-1.0 #zebra #giraffe #plains zebra #Grevy's zebra #video #animal behavior #behavior recognition #annotation #annotated video #conservation #drone #UAV #imbalanced #Kenya #Mpala Research Centre #doi-10.57967/hf/1010 #region-us 
 ### Dataset Summary


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
The drone footage is captured at a resolution of 5472 x 3078 pixels, and the videos were recorded at a frame rate of 29.97 frames per second. ### Supported Tasks and Leaderboards


The results of our evaluation using I3D, SlowFast, and X3D architectures are given in the table below. For each one, the model was trained for 120 epochs with batch size of 5. For more information on these results, see our paper. ### Languages


English


Dataset Structure
-----------------


Under 'KABR/dataset/image/', the data has been archived into '.zip' files, which are split into 2GB files. These must be recombined and extracted.
After cloning and navigating into the repository, you can use the following commands to do the reconstruction:


Alternatively, there is a download script, 'URL', which allows a download of the entire dataset in its established format without requiring one to clone the repository (cloning requires *at least* double the size of the dataset to store). To proceed with this approach, download 'URL' to the system where you want to access the data.
Then, in the same directory as the script, run the following to begin the download:


This script then downloads all the files present in the repository (without making a clone of the '.git' directory, etc.), concatenates the part files to their ZIP archives, verifies the MD5 checksums, extracts, and cleans up so that the folder structure, as described below, is present.


Note that it will require approximately 116GB of free space to complete this process, though the final dataset will only take about 61GB of disk space (the script removes the extra files after checking the download was successful).


The KABR dataset follows the Charades format:


The dataset can be directly loaded and processed by the SlowFast framework.


Informational Files


* 'KABR/configs': examples of SlowFast framework configs.
* 'KABR/annotation/URL': distribution of classes for all videos.


Scripts:


* 'URL': Encode image sequences into the original video.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'video/G0067.mp4'.
* 'URL': Encode image sequences into the original video with corresponding annotations.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'visual/G0067.mp4'. ### Data Instances


Naming: Within the image folder, the 'video\_n' folders are named as follows (X indicates a number):


* G0XXX.X - Giraffes
* ZP0XXX.X - Plains Zebras
* ZG0XXX.X - Grevy's Zebras
* Within each of these folders the images are simply 'X.jpg'.


Note: The dataset consists of a total of 1,139,893 frames captured from drone videos. There are 488,638 frames of Grevy's zebras, 492,507 frames of plains zebras, and 158,748 frames of giraffes. ### Data Fields


There are 14,764 unique behavioral sequences in the dataset. These consist of eight distinct behaviors:


* Walk
* Trot
* Run: animal is moving at a cantor or gallop
* Graze: animal is eating grass or other vegetation
* Browse: animal is eating trees or bushes
* Head Up: animal is looking around or observe surroundings
* Auto-Groom: animal is grooming itself (licking, scratching, or rubbing)
* Occluded: animal is not fully visible ### Data Splits


Training and validation sets are indicated by their respective CSV files ('URL' and 'URL'), located within the 'annotation' folder.


Dataset Creation
---------------- ### Curation Rationale


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
We believe that this dataset will be a valuable resource for researchers working on animal behavior recognition, as it provides a diverse and high-quality set of annotated videos that can be used for evaluating deep learning models.
Additionally, the dataset can be used to study the behavior patterns of Kenyan animals and can help to inform conservation efforts and wildlife management strategies.


We provide a detailed description of the dataset and its annotation process, along with some initial experiments on the dataset using conventional deep learning models.
The results demonstrate the effectiveness of the dataset for animal behavior recognition and highlight the potential for further research in this area. ### Source Data #### Initial Data Collection and Normalization


Data was collected from 6 January 2023 through 21 January 2023 at the Mpala Research Centre in Kenya under a Nacosti research license. We used DJI Mavic 2S drones equipped with cameras to record 5.4K resolution videos (5472 x 3078 pixels) from varying altitudes and distances of 10 to 50 meters from the animals (distance was determined by circumstances and safety regulations).


Mini-scenes were extracted from these videos to reduce the impact of drone movement and facilitate human annotation. Animals were detected in frame using YOLOv8, then the SORT tracking algorithm was applied to follow their movement. A 400 by 300 pixel window, centered on the animal, was then extracted; this is the mini-scene. ### Annotations #### Annotation process


In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations. ### Personal and Sensitive Information


Though there are endangered species included in this data, exact locations are not provided and their safety is assured by their location within the preserve.


Considerations for Using the Data
--------------------------------- ### Other Known Limitations


This data exhibits a long-tailed distribution due to the natural variation in frequency of the observed behaviors.


Additional Information
---------------------- ### Authors


* Maksim Kholiavchenko (Rensselaer Polytechnic Institute) - ORCID: 0000-0001-6757-1957
* Jenna Kline (The Ohio State University)
* Michelle Ramirez (The Ohio State University)
* Sam Stevens (The Ohio State University)
* Alec Sheets (The Ohio State University) - ORCID: 0000-0002-3737-1484
* Reshma Ramesh Babu (The Ohio State University) - ORCID: 0000-0002-2517-5347
* Namrata Banerji (The Ohio State University) - ORCID: 0000-0001-6813-0010
* Elizabeth Campolongo (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0846-2413
* Matthew Thompson (Imageomics Institute, The Ohio State University) - ORCID: 0000-0003-0583-8585
* Nina Van Tiel (Eidgenössische Technische Hochschule Zürich) - ORCID: 0000-0001-6393-5629
* Jackson Miliko (Mpala Research Centre)
* Eduardo Bessa (Universidade de Brasília) - ORCID: 0000-0003-0606-5860
* Tanya Berger-Wolf (The Ohio State University) - ORCID: 0000-0001-7610-1412
* Daniel Rubenstein (Princeton University) - ORCID: 0000-0001-9049-5219
* Charles Stewart (Rensselaer Polytechnic Institute) ### Licensing Information


This dataset is dedicated to the public domain for the benefit of scientific pursuits. We ask that you cite the dataset and journal paper using the below citations if you make use of it in your research. #### Dataset #### Paper ### Contributions


The Imageomics Institute is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
passage: TAGS
#task_categories-video-classification #size_categories-1M<n<10M #license-cc0-1.0 #zebra #giraffe #plains zebra #Grevy's zebra #video #animal behavior #behavior recognition #annotation #annotated video #conservation #drone #UAV #imbalanced #Kenya #Mpala Research Centre #doi-10.57967/hf/1010 #region-us 
### Dataset Summary


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
The drone footage is captured at a resolution of 5472 x 3078 pixels, and the videos were recorded at a frame rate of 29.97 frames per second.### Supported Tasks and Leaderboards


The results of our evaluation using I3D, SlowFast, and X3D architectures are given in the table below. For each one, the model was trained for 120 epochs with batch size of 5. For more information on these results, see our paper. passage: ### Languages


English


Dataset Structure
-----------------


Under 'KABR/dataset/image/', the data has been archived into '.zip' files, which are split into 2GB files. These must be recombined and extracted.
After cloning and navigating into the repository, you can use the following commands to do the reconstruction:


Alternatively, there is a download script, 'URL', which allows a download of the entire dataset in its established format without requiring one to clone the repository (cloning requires *at least* double the size of the dataset to store). To proceed with this approach, download 'URL' to the system where you want to access the data.
Then, in the same directory as the script, run the following to begin the download:


This script then downloads all the files present in the repository (without making a clone of the '.git' directory, etc.), concatenates the part files to their ZIP archives, verifies the MD5 checksums, extracts, and cleans up so that the folder structure, as described below, is present.


Note that it will require approximately 116GB of free space to complete this process, though the final dataset will only take about 61GB of disk space (the script removes the extra files after checking the download was successful).


The KABR dataset follows the Charades format:


The dataset can be directly loaded and processed by the SlowFast framework.


Informational Files


* 'KABR/configs': examples of SlowFast framework configs.
* 'KABR/annotation/URL': distribution of classes for all videos.


Scripts:


* 'URL': Encode image sequences into the original video.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'video/G0067.mp4'.
* 'URL': Encode image sequences into the original video with corresponding annotations.
	+ For example, '[image/G0067.1, image/G0067.2, ..., image/G0067.24]' will be encoded into 'visual/G0067.mp4'.### Data Instances


Naming: Within the image folder, the 'video\_n' folders are named as follows (X indicates a number):


* G0XXX.X - Giraffes
* ZP0XXX.X - Plains Zebras
* ZG0XXX.X - Grevy's Zebras
* Within each of these folders the images are simply 'X.jpg'.


Note: The dataset consists of a total of 1,139,893 frames captured from drone videos. There are 488,638 frames of Grevy's zebras, 492,507 frames of plains zebras, and 158,748 frames of giraffes.### Data Fields


There are 14,764 unique behavioral sequences in the dataset. These consist of eight distinct behaviors:


* Walk
* Trot
* Run: animal is moving at a cantor or gallop
* Graze: animal is eating grass or other vegetation
* Browse: animal is eating trees or bushes
* Head Up: animal is looking around or observe surroundings
* Auto-Groom: animal is grooming itself (licking, scratching, or rubbing)
* Occluded: animal is not fully visible### Data Splits


Training and validation sets are indicated by their respective CSV files ('URL' and 'URL'), located within the 'annotation' folder.


Dataset Creation
---------------- passage: ### Curation Rationale


We present a novel high-quality dataset for animal behavior recognition from drone videos.
The dataset is focused on Kenyan wildlife and contains behaviors of giraffes, plains zebras, and Grevy's zebras.
The dataset consists of more than 10 hours of annotated videos, and it includes eight different classes, encompassing seven types of animal behavior and an additional category for occluded instances.
In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.
The dataset was collected using drones that flew over the animals in the Mpala Research Centre in Kenya, providing high-quality video footage of the animal's natural behaviors.
We believe that this dataset will be a valuable resource for researchers working on animal behavior recognition, as it provides a diverse and high-quality set of annotated videos that can be used for evaluating deep learning models.
Additionally, the dataset can be used to study the behavior patterns of Kenyan animals and can help to inform conservation efforts and wildlife management strategies.


We provide a detailed description of the dataset and its annotation process, along with some initial experiments on the dataset using conventional deep learning models.
The results demonstrate the effectiveness of the dataset for animal behavior recognition and highlight the potential for further research in this area.### Source Data#### Initial Data Collection and Normalization


Data was collected from 6 January 2023 through 21 January 2023 at the Mpala Research Centre in Kenya under a Nacosti research license. We used DJI Mavic 2S drones equipped with cameras to record 5.4K resolution videos (5472 x 3078 pixels) from varying altitudes and distances of 10 to 50 meters from the animals (distance was determined by circumstances and safety regulations).


Mini-scenes were extracted from these videos to reduce the impact of drone movement and facilitate human annotation. Animals were detected in frame using YOLOv8, then the SORT tracking algorithm was applied to follow their movement. A 400 by 300 pixel window, centered on the animal, was then extracted; this is the mini-scene.### Annotations#### Annotation process


In the annotation process for this dataset, a team of 10 people was involved, with an expert zoologist overseeing the process.
Each behavior was labeled based on its distinctive features, using a standardized set of criteria to ensure consistency and accuracy across the annotations.### Personal and Sensitive Information


Though there are endangered species included in this data, exact locations are not provided and their safety is assured by their location within the preserve.


Considerations for Using the Data
---------------------------------### Other Known Limitations


This data exhibits a long-tailed distribution due to the natural variation in frequency of the observed behaviors.


Additional Information
----------------------