|
Can someone provide a detailed technical explanation of this concept in deep learning and transformer systems? |
Answered by
Ultron09
Aug 27, 2026
Replies: 1 comment
|
Cross-Entropy H(p, q) is mathematically equivalent to the entropy of the true distribution H(p) plus the KL divergence between true and predicted distributions D_KL(p || q): H(p, q) = H(p) + D_KL(p || q). Minimizing cross-entropy is identical to minimizing KL divergence from empirical ground truth. |
0 replies
Answer selected by
Ultron09
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Cross-Entropy H(p, q) is mathematically equivalent to the entropy of the true distribution H(p) plus the KL divergence between true and predicted distributions D_KL(p || q): H(p, q) = H(p) + D_KL(p || q). Minimizing cross-entropy is identical to minimizing KL divergence from empirical ground truth.