text
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
new_line
new_line
text
The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
new_line
new_line
text
Had claude make me a script, using the new Reflection-70B, and these are the results:
Had claude make me a script, using the new Reflection-70B, and these are the results:
new_line
new_line
text
Total weights: 70553706496
Total weights: 70553706496
new_line
text
Fully representable: 70530215524
Fully representable: 70530215524
new_line
text
Squashed: 23490972
Squashed: 23490972
new_line
text
Percentage squashed: 0.03%
Percentage squashed: 0.03%
new_line
new_line
text
0.03%!!!!
0.03%!!!!
new_line
new_line
text
A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
new_line
new_line
text
This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
new_line
new_line
text
Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)

The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16

Had claude make me a script, using the new Reflection-70B, and these are the results:

Total weights: 70553706496
Fully representable: 70530215524
Squashed: 23490972
Percentage squashed: 0.03%

0.03%!!!!

A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)

This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.

Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
🔥
prithivMLmods Sri-Vigneshwar-DJ Joseph717171 not-lain John6666 morph3v5
👀
John6666 Joseph717171
🤯
louisbrulenaudet Joseph717171
❤️
Joseph717171 KhaldiAbderrhmane
🤗
Joseph717171
🧠
Joseph717171
🚀
Joseph717171
👍
MoonRide
/avatars/da52e5fce67042332fa1e9f5fd3e5635.svg
Luke Chadwick
vertis
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6435718aaaef013d1aec3b8b/XKf-8MA47tjVAM6SCX0MP.jpeg
Bartowski
bartowski
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6640bbd0220cfa8cbfdce080/wiAHUu5ewawyipNs0YFBR.png
John Smith
John6666
user
/avatars/ea4398745974d781ae9dc0e95b12cabe.svg
Joseph
Joseph717171
user
/avatars/99351620d65d263418e6d0d4e170f055.svg
Abrosimov
ajiriro
user
https://cdn-avatars.huggingface.co/v1/production/uploads/66d1efa935c36f266f507cff/a2-fPLeGwAp5fqCKdqfzp.jpeg
Harmendo
Hampetiudo
user
/avatars/3b03217c22442b7bfed9beac2bf50d17.svg
Alex Daminger
Handgun1773
user
/avatars/98d7cbc7bf4cbf4f2810cbc0a1a34d64.svg
Iwan Kawrakow
ikawrakow
user
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4Az8a8F60rNOD3L3ThsCe.png
Compilade
compilade
user