Cx/instruction level acc sim by firemountain154B · Pull Request #15 · AICrossSim/PLENA

firemountain154B · 2025-07-14T21:05:57Z

No description provided.

jiaeenie · 2025-07-14T21:35:17Z

Would it be good to ignore this file?

jiaeenie · 2025-07-14T22:06:13Z

A general question: why do we quantize the outputs of matmul, RMSNorm, softmax, and SiLU and should we also include the output of Embedding layer, and ROPE to minifloat here? Is it to simulate the fact that activations are written to Vector Core SRAM in minifloat, and then later fetched and cast to MXFP as input(which we had so far)? Would @ChengZhang-98 be able to help me to validate if it is the correct behaviour in the simulator? Thanks a lot!

jiaeenie · 2025-07-14T21:38:07Z

        enable_eval_harness: Whether to run evaluation via EleutherAI lm-eval-harness.
    """
-    preset_mxfp_X, preset_mxfp_W, preset_mxfp_Kv, preset_minifloat_NL = validate_and_sanitize_quant_args(
+    preset_mxfp_X, preset_mxfp_W, preset_mxfp_Kv, preset_minifloat_X, preset_minifloat_NL = validate_and_sanitize_quant_args(


Why do we introduce a new flag here, I think preset_minifloat_NL here sets the minifloat format for all nonlinear ops.

the minifloat_x is actually not the minifloat_NL, I think this behaviour should be merged into X quant, cause in hardware, if you have a matmul after matmul (projection after ov in attention), the intermediate result will also be stored in vector sram which is minifloat.

jiaeenie · 2025-07-14T21:45:08Z

    if func_type =="Xq":
        assert x_minifp_meta is not None, "MinifloatMeta must be provided for 'Xq' input"
-        input = minifloat_ieee_quantizer(input, x_minifp_meta)
+        quantizer = partial(minifloat_ieee_quantizer, meta=x_minifp_meta)


Shall we make the hardware-aware quantization optional in the simulation framework, while keeping the previous coarser-grained input quantization as the default?

to be honest, this is not that hardware related, this is instruction-aware quantization (I didn't expose those tiny hardware mismatch in this code), so in this sense, if we just quantize the input and output of non-linear, the acc will have a really big accuracy mismatch with real hardware, which unacceptable I guess

jiaeenie · 2025-07-14T21:48:25Z

@@ -0,0 +1,187 @@
+from typing import Literal, Union


Could see the new changes in main for config parsing.

sorry my bad, the new changes looks better, but can we include the preset_minifloat_X stuff in the the config parsing

jiaeenie · 2025-07-14T21:49:44Z

@@ -10,14 +10,15 @@ def _minifloat_ieee_quantize(x: Tensor, meta: MinifloatMeta) -> Tensor:
    mantissa_bits = meta.element_frac_bits
    exponent_bias = meta.exponent_bias



I need some help here, would it be okay for @ChengZhang-98 to review this file?

Note here: the -1 to -2 is for describing the hardware behaviour, we're still leaving one bit for inf.

the later one is to handle the situation when we set config to higher bits, the quantizer will turn inf to nan

May I ask why we keep Inf instead of saturation for minifloat in hardware?

this is related to the hardware design itself, in 4 bit exponent case, bias is 4'b0111, which represents 7, the inf is 1111,
while doing some operation on exponent, the current design will take exponent bit out and tansfer it to the unbiased exponent. the unbiased exponent will be a signed integer.

In the inf case, 1111 - 0111 = 1000, if we still want to operate in 4 bit, the signed 4'b1000 actually means -8 ( but we want it to be 8).
There are several design choices to deal with this, for example, rewrite all the addition, or increase one bit. but in this case, we choose to keep the inf bit, cause 1. this is for minifloat(we're possible to have more bit on this), we can have another version for lower bit-width case (mxfp), 2. to align with IEEE standard. 3. don't want to break the design.

jiaeenie · 2025-07-14T22:25:53Z

A question here, it doesn't seem like the hardware_aware_operations are exposing dse precision related parameters such as FP_ADD_EXT_EXP_WIDTH etc. More specifically the ones defined in this config file here.
@GeorgeWu1204 Please correct me if I'm wrong

Yes, please just ignore these parameters for now; I will update them later.

So in this, I'd say this is not that hardware related, just instruction aware

ChengZhang-98 · 2025-07-15T17:34:17Z

@@ -10,14 +10,15 @@ def _minifloat_ieee_quantize(x: Tensor, meta: MinifloatMeta) -> Tensor:
    mantissa_bits = meta.element_frac_bits
    exponent_bias = meta.exponent_bias



May I ask why we keep Inf instead of saturation for minifloat in hardware?

ChengZhang-98 · 2025-07-15T17:40:04Z

+        #     f"Legal values are: {legal_scale_exp_bits}."
+        # )

-        legal_element_exp_frac_bits = ((4, 3), (5, 2), (2, 3), (3, 2), (2, 1))


This is dangerous. The MXFP quantizer in mase-triton does not aim to support all the possible (scale bit, exp bit, frac bit) setups, which is impossible for the limited number of dtypes in pytorch/GPU.

Current quantizer in mase-triton assumes the input data is BF16. FP32/FP64 numbers will be cast to BF16 implicitly before converting to MXFP.

ChengZhang-98 · 2025-07-15T17:41:29Z

+)
+import re
+
+# 提取 m4e3 中的 4 和 3


Remove this, or use Eng.

What does cc in the file name mean?

some historical reason,, jiayi made a new set_quant_args, but I'm not awaring this, so we might want to keep jiayi's new quant_args, so then random put this file here.. will be removed later

ChengZhang-98 · 2025-07-15T17:47:53Z

        enable_eval_harness: Whether to run evaluation via EleutherAI lm-eval-harness.
    """
-    preset_mxfp_X, preset_mxfp_W, preset_mxfp_Kv, preset_minifloat_NL = validate_and_sanitize_quant_args(
+    preset_mxfp_X, preset_mxfp_W, preset_mxfp_Kv, preset_minifloat_X, preset_minifloat_NL = validate_and_sanitize_quant_args(


Could someone elaborate the difference between preset_minifloat_X and preet_minifloat_NL?
Do we need both?

btw, I think docstrings are needed to explain the meaning of these notations.

in terms of naming, probably N is better than NL? We use KV because that's two tensors, key and value, if we want, we can support separate precisions for K and V, but NL is just non-linear tensor?

ChengZhang-98 · 2025-07-15T17:53:20Z

+    OCP_MXFP8_E4M3,
+    OCP_MXFP8_E5M2,
+)
+import re


probably move this import of built-in package to the top of the file

ChengZhang-98 · 2025-07-15T17:53:59Z

@@ -0,0 +1,13 @@
+


Is this related to acc-sim?

will be removed

firemountain154B added 8 commits July 12, 2025 22:16

non-linears

71c2c6f

I don't wnat loss connection again

53ecc62

find a bug

15befa9

fix e8m7 error, pytorch is built on e5m10

f407069

refining the hardware related accuracy simulator

968323f

merge new

310fe8f

support acc_sim

efa7b23

instruction aware mxfp

f64658a

firemountain154B requested review from GeorgeWu1204 and jiaeenie July 14, 2025 21:05

jiaeenie reviewed Jul 14, 2025

View reviewed changes

jiaeenie requested a review from ChengZhang-98 July 14, 2025 22:06

jiaeenie requested changes Jul 14, 2025

View reviewed changes

firemountain154B requested a review from jiaeenie July 15, 2025 08:46

ChengZhang-98 reviewed Jul 15, 2025

View reviewed changes

		@@ -10,14 +10,15 @@ def _minifloat_ieee_quantize(x: Tensor, meta: MinifloatMeta) -> Tensor:
		mantissa_bits = meta.element_frac_bits
		exponent_bias = meta.exponent_bias

Uh oh!

Conversation

firemountain154B commented Jul 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiaeenie commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

firemountain154B Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

firemountain154B Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jiaeenie commented Jul 14, 2025 •

edited

Loading

firemountain154B Jul 15, 2025 •

edited

Loading

firemountain154B Jul 15, 2025 •

edited

Loading