Floating Data Types

Here are the commonly used floating point data types choice of which impacts both memory usage and throughput:

• fp32 (float32)
• fp16 (float16)
• bf16 (bfloat16)
• tf32 (CUDA internal data type)

2. fp16

3. Related

4. References

• Performance and Scalability: How To Fit a Bigger Model and Train It Faster