Cleaned dataset for experiments and fast pretokenizing. The SQL query to create the dataset is the following:
```python
SELECT
  f.repo_name,
  f.path,
  c.copies,
  c.size,
  c.content,
  l.license
FROM
 (select f.*, row_number() over (partition by id order by path desc) as seqnum
      from `bigquery-public-data.github_repos.files` AS f) f
JOIN
  `bigquery-public-data.github_repos.contents` AS c
ON
  f.id = c.id AND seqnum=1
JOIN
  `bigquery-public-data.github_repos.licenses` AS l
ON
  f.repo_name = l.repo_name
WHERE
  NOT c.binary
  AND ((f.path LIKE '%.vhdl'
      OR f.path LIKE '%.vhd'
    AND (c.size BETWEEN 0
      AND 1048575)))
```
license:apache-2.0 region:us
{"license": "apache-2.0"}
TAGS
#license-apache-2.0 #region-us 

Cleaned dataset for experiments and fast pretokenizing. The SQL query to create the dataset is the following:

TAGS
#license-apache-2.0 #region-us 

passage: TAGS
#license-apache-2.0 #region-us 
