text
Today lets discuss about 32-bit (FP32) and 16-bit (FP16) floating-point!
Today lets discuss about 32-bit (FP32) and 16-bit (FP16) floating-point!
new_line
new_line
text
Floating-point numbers are used to represent real numbers (like decimals) and they consist of three parts:
Floating-point numbers are used to represent real numbers (like decimals) and they consist of three parts:
new_line
new_line
code_fence
```
Sign bit: 
Indicates whether the number is positive (0) or negative (1).
Exponent:
Determines the scale of the number (i.e., how large or small it is by shifting the decimal point).
Mantissa (or fraction): 
Represents the actual digits of the number.
```
Sign bit: 
Indicates whether the number is positive (0) or negative (1).
Exponent:
Determines the scale of the number (i.e., how large or small it is by shifting the decimal point).
Mantissa (or fraction): 
Represents the actual digits of the number.
new_line
text
32-bit Floating Point (FP32)
32-bit Floating Point (FP32)
new_line
text
Total bits: 32 bits
Total bits: 32 bits
new_line
text
Sign bit: 1 bit
Sign bit: 1 bit
new_line
text
Exponent: 8 bits
Exponent: 8 bits
new_line
text
Mantissa: 23 bits
Mantissa: 23 bits
new_line
text
For example:
For example:
new_line
text
A number like -15.375 would be represented as:
A number like -15.375 would be represented as:
new_line
text
Sign bit: 1 (negative number)
Sign bit: 1 (negative number)
new_line
text
Exponent: Stored after being adjusted by a bias (127 in FP32).
Exponent: Stored after being adjusted by a bias (127 in FP32).
new_line
text
Mantissa: The significant digits after converting the number to binary.
Mantissa: The significant digits after converting the number to binary.
new_line
new_line
text
16-bit Floating Point (FP16)
16-bit Floating Point (FP16)
new_line
text
Total bits: 16 bits
Total bits: 16 bits
new_line
text
Sign bit: 1 bit
Sign bit: 1 bit
new_line
text
Exponent: 5 bits
Exponent: 5 bits
new_line
text
Mantissa: 10 bits
Mantissa: 10 bits
new_line
text
Example:
Example:
new_line
text
A number like -15.375 would be stored similarly:
A number like -15.375 would be stored similarly:
new_line
text
Sign bit: 1 (negative number)
Sign bit: 1 (negative number)
new_line
text
Exponent: Uses 5 bits, limiting the range compared to FP32.
Exponent: Uses 5 bits, limiting the range compared to FP32.
new_line
text
Mantissa: Only 10 bits for precision.
Mantissa: Only 10 bits for precision.
new_line
new_line
text
Precision and Range
Precision and Range
new_line
text
FP32: Higher precision and larger range, with about 7 decimal places of accuracy.
FP32: Higher precision and larger range, with about 7 decimal places of accuracy.
new_line
text
FP16: Less precision (around 3-4 decimal places), smaller range but faster computations and less memory use.
FP16: Less precision (around 3-4 decimal places), smaller range but faster computations and less memory use.
Today lets discuss about 32-bit (FP32) and 16-bit (FP16) floating-point!

Floating-point numbers are used to represent real numbers (like decimals) and they consist of three parts:

```
Sign bit: 
Indicates whether the number is positive (0) or negative (1).
Exponent:
Determines the scale of the number (i.e., how large or small it is by shifting the decimal point).
Mantissa (or fraction): 
Represents the actual digits of the number.
```
32-bit Floating Point (FP32)
Total bits: 32 bits
Sign bit: 1 bit
Exponent: 8 bits
Mantissa: 23 bits
For example:
A number like -15.375 would be represented as:
Sign bit: 1 (negative number)
Exponent: Stored after being adjusted by a bias (127 in FP32).
Mantissa: The significant digits after converting the number to binary.

16-bit Floating Point (FP16)
Total bits: 16 bits
Sign bit: 1 bit
Exponent: 5 bits
Mantissa: 10 bits
Example:
A number like -15.375 would be stored similarly:
Sign bit: 1 (negative number)
Exponent: Uses 5 bits, limiting the range compared to FP32.
Mantissa: Only 10 bits for precision.

Precision and Range
FP32: Higher precision and larger range, with about 7 decimal places of accuracy.
FP16: Less precision (around 3-4 decimal places), smaller range but faster computations and less memory use.
🔥
YaTharThShaRma999 John6666 lucianosb victor Gausq
🧠
prithivMLmods lucianosb
https://cdn-avatars.huggingface.co/v1/production/uploads/6640bbd0220cfa8cbfdce080/wiAHUu5ewawyipNs0YFBR.png
John Smith
John6666
user
/avatars/5cbfa6cbde933503bbc3577cf713e7b5.svg
Friedrich Marty
Smorty100
user
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/y1W6Co4jIYB95Cx6Tjrsd.jpeg
Muhammad Imran Zaman
ImranzamanML
user