Greedy Sampling

[luisfrentzen-cc]

Is there any way to make sure that we use greedy sampling? I don't find do_sample parameter anywhere.

[turboderp]

You can sample with top-K = 1 for greedy sampling.

[nvlas]

Hi. Reopening this, as I have trouble getting the greedy sampling to work as expected. Shouldn't this code produce identical results?

c = ExLlamaV2Sampler.Settings.greedy()
n = 5
outputs = generator.generate(
        prompt = ["Once upon a time"]*n,
        gen_settings = [c]*n,
        max_new_tokens = 20,
    )
for out in outputs:
    print(out)

I get things like this:

Once upon a time, in a small village nestled in the rolling hills of the countryside, there lived a young girl named
Once upon a time, there was a small village nestled in the heart of a dense forest. The villagers lived simple lives
Once upon a time, in a small village nestled in the rolling hills of the countryside, there lived a young girl named
Once upon a time, there was a small village nestled in the heart of a dense forest. The villagers lived simple lives
Once upon a time, there was a small village nestled in the heart of a dense forest. The villagers lived simple lives

[nvlas]

apparently, the extra option add_bos = True is needed to get greedy to work...

[turboderp]

Whether you should include BOS or not is dependent on the model. But in any case greedy sampling doesn't guarantee perfect determinism.

[nvlas]

Is there some way to guarantee determinism? (Is the lack of determinism specific to exllamav2?)

[turboderp]

Short answer, no.

Some frameworks are more deterministic than others (and ExLlama is one of the less deterministic ones), but in the strictest sense determinism is impossible to guarantee in any framework without seriously hindering performance. For an illustration, you can try this plain PyTorch example to see the effect of batch size on otherwise mathematically equivalent operations:

import torch

def test_pair(a, b, dtype, device):
    a = a.to(dtype).to(device)
    b = b.to(dtype).to(device)
    for bsz in [1, 2, 4, 8, 16, 32, 64]:
        # Create a batch of vectors
        c = torch.cat([a] * bsz, dim = 0)
        # Multiply the identical vectors in the batch with the matrix
        d = c @ b
        # Compute the sum of the first entry in the output batch
        e = d[0].sum().item()
        print(f"device={str(a.device):<6}  dtype={str(a.dtype):<13} bsz={bsz:<3} sum: {e:.12f}")

torch.manual_seed(0)
a = torch.randn((1, 1024), dtype = torch.half)
b = torch.randn((1024, 1024), dtype = torch.half)

test_pair(a, b, torch.half, "cpu")
test_pair(a, b, torch.float, "cpu")
test_pair(a, b, torch.half, "cuda:0")
test_pair(a, b, torch.float, "cuda:0")

In my case, I even get different results between different NVIDIA GPUs. Here's float32 on a 4090:

device=cuda:0  dtype=torch.float32 bsz=1   sum: 42.869323730469
device=cuda:0  dtype=torch.float32 bsz=2   sum: 42.869720458984
device=cuda:0  dtype=torch.float32 bsz=4   sum: 42.869720458984
device=cuda:0  dtype=torch.float32 bsz=8   sum: 42.869720458984
device=cuda:0  dtype=torch.float32 bsz=16  sum: 42.869720458984
device=cuda:0  dtype=torch.float32 bsz=32  sum: 42.869476318359
device=cuda:0  dtype=torch.float32 bsz=64  sum: 42.869537353516

vs a 3090:

device=cuda:3  dtype=torch.float32 bsz=1   sum: 42.869323730469
device=cuda:3  dtype=torch.float32 bsz=2   sum: 42.869659423828
device=cuda:3  dtype=torch.float32 bsz=4   sum: 42.869659423828
device=cuda:3  dtype=torch.float32 bsz=8   sum: 42.869659423828
device=cuda:3  dtype=torch.float32 bsz=16  sum: 42.869659423828
device=cuda:3  dtype=torch.float32 bsz=32  sum: 42.869415283203
device=cuda:3  dtype=torch.float32 bsz=64  sum: 42.869598388672

CUDA guarantees that results are mathematically correct, not that they're numerically stable across batch sizes, or even between different driver versions. It also offers some tools that you can optionally use to trade more determinism for more performance, such as reduction by atomic addition, and ExLlama makes use of those, for performance.

My take is generally that determinism is a red herring, and if you're designing something that relies on it, then there's an inherent problem with that design. You can end up with software that works one day, and then fails after a driver update, or fails when you pass it off to someone else to test on different hardware, or if you try to scale it up with concurrency, or whatever.

I expect the next version (i.e. ExLlamaV3) to be somewhat more deterministic, making different tradeoffs, but it's important to understand that there are some hard limits imposed by CUDA in the end. With ExLlamaV2 as it is, the best you could probably do is use an FP16 model, since that at least avoids atomic additions.