Alusus language bindings for llama.cpp, providing a complete interface for running LLM inference locally. This library supports both CPU and Vulkan GPU backends.
import "Apm";
Apm.importPackage("Alusus/Llama@0.2");
import "Srl/Console";
import "Apm";
Apm.importPackage("Alusus/Llama@0.2");
use Srl;
use Llama;
// Load backends
Ggml.Backend.cpuLoad();
Ggml.Backend.vkLoad(); // Optional: for Vulkan GPU support
// Load model
def model: ref[Model](Model.load("model.gguf", Model.getDefaultParams()));
def ctx: ref[Context](Context.initFromModel(model, Context.getDefaultParams()));
// Tokenize input
def tokens: array[Token, 512];
def nTokens: Int = tokenize(model.vocab, "Hello", 5, tokens, 512, true, true);
// Decode and generate
def batch: Batch = Batch.getOne(tokens, nTokens);
ctx.decode(batch);
// Sample next token
def sampler: ref[Sampler](Sampler.initGreedy());
def nextToken: Token = sampler.sample(ctx, -1);
// Cleanup
Sampler.free(sampler);
Context.free(ctx);
Model.free(model);
Set the GGML_USE_VULKAN environment variable to 1 before running to enable Vulkan GPU acceleration:
GGML_USE_VULKAN=1 alusus your_script.alusus
Token:Int- Token ID (alias forllama_tokenin llama.cpp)Pos:Int- Token position (alias forllama_posin llama.cpp)SeqId:Int- Sequence ID (alias forllama_seq_idin llama.cpp)ProgressCallback:ptr[function (progress: Float, userData: ptr): Bool]- Progress callback function pointer (alias forllama_progress_callbackin llama.cpp)
DEFAULT_SEED(0xFFFFFFFF): Default random seed (equivalent toLLAMA_DEFAULT_SEEDin llama.cpp)TOKEN_NULL(-1): Null token value (equivalent toLLAMA_TOKEN_NULLin llama.cpp)
Controls how model layers are distributed across GPUs.
NONE: Single GPU only (equivalent toLLAMA_SPLIT_MODE_NONEin llama.cpp)LAYER: Split layers and KV cache across GPUs (equivalent toLLAMA_SPLIT_MODE_LAYERin llama.cpp)ROW: Split with tensor parallelism if supported (equivalent toLLAMA_SPLIT_MODE_ROWin llama.cpp)
RoPE (Rotary Position Embedding) scaling type.
UNSPECIFIED: Not specified (equivalent toLLAMA_ROPE_SCALING_TYPE_UNSPECIFIEDin llama.cpp)NONE: No scaling (equivalent toLLAMA_ROPE_SCALING_TYPE_NONEin llama.cpp)LINEAR: Linear scaling (equivalent toLLAMA_ROPE_SCALING_TYPE_LINEARin llama.cpp)YARN: YaRN scaling (equivalent toLLAMA_ROPE_SCALING_TYPE_YARNin llama.cpp)LONGROPE: LongRoPE scaling (equivalent toLLAMA_ROPE_SCALING_TYPE_LONGROPEin llama.cpp)
Pooling type for embeddings.
UNSPECIFIED: Not specified (LLAMA_POOLING_TYPE_UNSPECIFIEDin llama.cpp)NONE: No pooling (LLAMA_POOLING_TYPE_NONEin llama.cpp)MEAN: Mean pooling (LLAMA_POOLING_TYPE_MEANin llama.cpp)CLS: CLS token pooling (LLAMA_POOLING_TYPE_CLSin llama.cpp)LAST: Last token pooling (LLAMA_POOLING_TYPE_LASTin llama.cpp)RANK: Rank pooling (LLAMA_POOLING_TYPE_RANKin llama.cpp)
Attention mechanism type.
UNSPECIFIED: Not specified (LLAMA_ATTENTION_TYPE_UNSPECIFIEDin llama.cpp)CAUSAL: Causal attention - autoregressive (LLAMA_ATTENTION_TYPE_CAUSALin llama.cpp)NON_CAUSAL: Non-causal attention (LLAMA_ATTENTION_TYPE_NON_CAUSALin llama.cpp)
Flash attention configuration.
AUTO(-1): Automatic selectionDISABLED(0): Flash attention disabledENABLED(1): Flash attention enabled
Token data with probability information. Maps to llama_token_data.
id(Token): Token ID. Maps toid.logit(Float): Log probability. Maps tologit.p(Float): Probability. Maps top.
Array of token data for sampling. Maps to llama_token_data_array.
data(ptr[TokenData]): Pointer to token data array. Maps todata.size(ArchWord): Number of elements. Maps tosize.selected(Int[64]): Selected token index. Maps toselected.sorted(Bool): Whether array is sorted. Maps tosorted.
Chat message structure for chat templates. Maps to llama_chat_message.
role(CharsPtr): Message role ("user", "assistant", "system"). Maps torole.content(CharsPtr): Message content. Maps tocontent.
Performance data for context operations. Maps to llama_perf_context_data.
tStartMs(Float[64]): Start time in milliseconds. Maps tot_start_ms.tLoadMs(Float[64]): Load time in milliseconds. Maps tot_load_ms.tPEvalMs(Float[64]): Prompt evaluation time. Maps tot_p_eval_ms.tEvalMs(Float[64]): Token evaluation time. Maps tot_eval_ms.nPEval(Int): Number of prompt evaluations. Maps ton_p_eval.nEval(Int): Number of token evaluations. Maps ton_eval.nReused(Int): Number of reused evaluations. Maps ton_reused.
Performance data for sampler operations. Maps to llama_perf_sampler_data.
tSampleMs(Float[64]): Sampling time in milliseconds. Maps tot_sample_ms.nSample(Int): Number of samples. Maps ton_sample.
Represents a loaded LLM model. Maps to llama_model.
Model loading parameters. Maps to llama_model_params.
devices(ptr): Device list. Maps todevices.tensorBuftOverrides(ref[array[TensorBuftOverride]]): Tensor buffer type overrides. Maps totensor_buft_overrides.nGpuLayers(Int): Number of layers to offload to GPU. Maps ton_gpu_layers.splitMode(SplitMode): GPU split mode. Maps tosplit_mode.mainGpu(Int): Main GPU index. Maps tomain_gpu.tensorSplit(ref[array[Float]]): Tensor split ratios. Maps totensor_split.progressCallback(ProgressCallback): Progress callback function. Maps toprogress_callback.progressCallbackUserData(ptr): User data for callback. Maps toprogress_callback_user_data.kvOverrides(ptr): KV cache overrides. Maps tokv_overrides.vocabOnly(Bool): Load vocabulary only. Maps tovocab_only.useMmap(Bool): Use memory mapping. Maps touse_mmap.useDirectIo(Bool): Use direct I/O. Maps touse_direct_io.useMlock(Bool): Lock memory. Maps touse_mlock.checkTensors(Bool): Validate tensor data. Maps tocheck_tensors.useExtraBufts(Bool): Use extra buffer types. Maps touse_extra_bufts.noHost(Bool): No host buffer allocation. Maps tono_host.noAlloc(Bool): No memory allocation. Maps tono_alloc.
func Model.getDefaultParams(): Params
Get default model parameters. Maps to llama_model_default_params.
func Model.load(path: CharsPtr, params: Params): ref[Model]
Load model from file. Maps to llama_model_load_from_file.
func Model.loadSplits(paths: ref[array[CharsPtr]], count: Word, params: Params): ref[Model]
Load model from split files. Maps to llama_model_load_from_splits.
func Model.free(model: ref[Model])
Free model resources. Maps to llama_model_free.
func model.save(path: CharsPtr)
Save model to file. Maps to llama_model_save_to_file.
model.hasEncoder: Bool
Check if model has encoder. Maps to llama_model_has_encoder.
model.hasDecoder: Bool
Check if model has decoder. Maps to llama_model_has_decoder.
model.nCtxTrain: Int
Training context size. Maps to llama_model_n_ctx_train.
model.nEmbd: Int
Embedding dimension. Maps to llama_model_n_embd.
model.nLayer: Int
Number of layers. Maps to llama_model_n_layer.
model.nHead: Int
Number of attention heads. Maps to llama_model_n_head.
model.vocab: ref[Vocab]
Get vocabulary. Maps to llama_model_get_vocab.
func model.metaValStr(key: CharsPtr, buf: CharsPtr, bufSize: ArchWord): Int
Get metadata value as string. Maps to llama_model_meta_val_str.
model.metaCount: Int
Get metadata count. Maps to llama_model_meta_count.
func model.metaKeyByIndex(idx: Int, buf: CharsPtr, bufSize: ArchWord): Int
Get metadata key by index. Maps to llama_model_meta_key_by_index.
Inference context for a model. Maps to llama_context.
Context creation parameters. Maps to llama_context_params.
nCtx(Word): Context size (0 = use model default). Maps ton_ctx.nBatch(Word): Logical batch size. Maps ton_batch.nUbatch(Word): Physical batch size. Maps ton_ubatch.nSeqMax(Word): Maximum sequences. Maps ton_seq_max.nThreads(Int): Number of threads for generation. Maps ton_threads.nThreadsBatch(Int): Number of threads for batch processing. Maps ton_threads_batch.ropeScalingType(RopeScalingType): RoPE scaling type. Maps torope_scaling_type.poolingType(PoolingType): Pooling type. Maps topooling_type.attentionType(AttentionType): Attention type. Maps toattention_type.flashAttnType(FlashAttnType): Flash attention setting. Maps toflash_attn.ropeFreqBase(Float): RoPE base frequency. Maps torope_freq_base.ropeFreqScale(Float): RoPE frequency scale. Maps torope_freq_scale.yarnExtFactor(Float): YaRN extrapolation factor. Maps toyarn_ext_factor.yarnAttnFactor(Float): YaRN attention factor. Maps toyarn_attn_factor.yarnBetaFast(Float): YaRN beta fast. Maps toyarn_beta_fast.yarnBetaSlow(Float): YaRN beta slow. Maps toyarn_beta_slow.yarnOrigCtx(Word): YaRN original context size. Maps toyarn_orig_ctx.defragThold(Float): Defragmentation threshold. Maps todefrag_thold.cbEval(function ptr): Evaluation callback. Maps tocb_eval.cbEvalData(ptr): Evaluation callback user data. Maps tocb_eval_user_data.typeK(Ggml.Type): K cache data type. Maps totype_k.typeV(Ggml.Type): V cache data type. Maps totype_v.abortCb(function ptr): Abort callback. Maps toabort_callback.abortCbData(ptr): Abort callback user data. Maps toabort_callback_data.embeddings(Bool): Enable embeddings output. Maps toembeddings.offloadKqv(Bool): Offload KQV to GPU. Maps tooffload_kqv.noPerf(Bool): Disable performance counters. Maps tono_perf.opOffload(Bool): Enable operation offloading. Maps toop_offload.swaFull(Bool): Full sliding window attention. Maps toswa_full.kvUnified(Bool): Unified KV cache. Maps tokv_unified.samplers(ref[array[SamplerSeqConfig]]): Per-sequence samplers. Maps tosamplers.nSamplers(ArchWord): Number of samplers. Maps ton_samplers.
func Context.getDefaultParams(): Params
Get default context parameters. Maps to llama_context_default_params.
func Context.initFromModel(model: ref[Model], params: Params): ref[Context]
Create context from model. Maps to llama_init_from_model.
func Context.free(ctx: ref[Context])
Free context resources. Maps to llama_free.
func ctx.attachThreadpool(tp: ref[Ggml.Threadpool], tpBatch: ref[Ggml.Threadpool])
Attach thread pool. Maps to llama_attach_threadpool.
func ctx.detachThreadpool()
Detach thread pool. Maps to llama_detach_threadpool.
func ctx.getEmbeddingsSeq(seqId: SeqId): ref[array[Float]]
Get sequence embeddings. Maps to llama_get_embeddings_seq.
func ctx.getModel(): ref[Model]
Get associated model. Maps to llama_get_model.
ctx.nCtx: Word
Get context size. Maps to llama_n_ctx.
ctx.nBatch: Word
Get logical batch size. Maps to llama_n_batch.
ctx.nUbatch: Word
Get physical batch size. Maps to llama_n_ubatch.
ctx.nSeqMax: Word
Get max sequences. Maps to llama_n_seq_max.
func ctx.getMemory(): ptr[Memory]
Get KV cache memory. Maps to llama_get_memory.
ctx.poolingType: PoolingType
Get pooling type. Maps to llama_pooling_type.
func ctx.encode(batch: Batch): Int
Encode batch (for encoder models). Maps to llama_encode.
func ctx.decode(batch: Batch): Int
Decode batch. Maps to llama_decode.
func ctx.getLogitsIth(i: Int): ref[array[Float]]
Get logits for token at index. Maps to llama_get_logits_ith.
func ctx.getEmbeddingsIth(i: Int): ref[array[Float]]
Get embeddings for token at index. Maps to llama_get_embeddings_ith.
ctx.stateSize: ArchWord
Get state size in bytes. Maps to llama_state_get_size.
func ctx.stateGetData(dst: ref[array[Word[8]]], size: ArchWord): ArchWord
Get state data. Maps to llama_state_get_data.
func ctx.stateSetData(src: ref[array[Word[8]]], size: ArchWord): ArchWord
Set state data. Maps to llama_state_set_data.
func ctx.stateLoadFile(path: CharsPtr, tokensOut: ref[array[Token]], cap: ArchWord, outCount: ref[ArchWord]): Bool
Load state from file. Maps to llama_state_load_file.
func ctx.stateSaveFile(path: CharsPtr, tokens: ref[array[Token]], count: ArchWord): Bool
Save state to file. Maps to llama_state_save_file.
func ctx.setAdapterLora(ad: ref[Adapter], scale: Float): Int
Apply LoRA adapter. Maps to llama_set_adapter_lora.
func ctx.removeAdapterLora(ad: ref[Adapter]): Int
Remove LoRA adapter. Maps to llama_rm_adapter_lora.
func ctx.clearAdapterLora()
Clear all LoRA adapters. Maps to llama_clear_adapter_lora.
ctx.perfContext: PerfContextData
Get performance data. Maps to llama_perf_context.
func ctx.perfContextPrint()
Print performance data. Maps to llama_perf_context_print.
func ctx.perfContextReset()
Reset performance counters. Maps to llama_perf_context_reset.
Token batch for processing. Maps to llama_batch.
nTokens(Int): Number of tokens. Maps ton_tokens.token(ref[array[Token]]): Token IDs. Maps totoken.embd(ref[array[Float]]): Embeddings (alternative to tokens). Maps toembd.pos(ref[array[Pos]]): Token positions. Maps topos.nSeqId(ref[array[Int]]): Number of sequence IDs per token. Maps ton_seq_id.seqId(ref[array[ref[array[SeqId]]]]): Sequence IDs. Maps toseq_id.output(ref[array[Int[8]]]): Output flags. Maps tologits.
func Batch.getOne(tokens: ref[array[Token]], nTokens: Int): Batch
Create batch from token array. Maps to llama_batch_get_one.
func Batch.init(nTokens: Int, embd: Int, nSeqMax: Int): Batch
Initialize empty batch. Maps to llama_batch_init.
func Batch.free(batch: Batch)
Free batch resources. Maps to llama_batch_free.
Token sampler for generation. Maps to llama_sampler.
Sampler chain parameters. Maps to llama_sampler_chain_params.
noPerf(Bool): Disable performance counters. Maps tono_perf.
func Sampler.init(iface: ptr, ctx: ptr): ref[Sampler]
Initialize custom sampler. Maps to llama_sampler_init.
func Sampler.chainInit(params: ChainParams): ref[Sampler]
Create sampler chain. Maps to llama_sampler_chain_init.
func Sampler.initGreedy(): ref[Sampler]
Create greedy sampler. Maps to llama_sampler_init_greedy.
func Sampler.initDist(seed: Word): ref[Sampler]
Create distribution sampler. Maps to llama_sampler_init_dist.
func Sampler.initTopK(k: Int): ref[Sampler]
Create top-k sampler. Maps to llama_sampler_init_top_k.
func Sampler.initTopP(p: Float, minKeep: ArchWord): ref[Sampler]
Create top-p (nucleus) sampler. Maps to llama_sampler_init_top_p.
func Sampler.initPenalties(penaltyLastN: Int, penaltyRepeat: Float, penaltyFreq: Float, penaltyPresent: Float): ref[Sampler]
Create penalty sampler. Maps to llama_sampler_init_penalties.
penaltyLastN: Last n tokens to penalize (0 = disable, -1 = context size)penaltyRepeat: Repeat penalty (1.0 = disabled)penaltyFreq: Frequency penalty (0.0 = disabled)penaltyPresent: Presence penalty (0.0 = disabled)
func Sampler.clone(s: ref[Sampler]): ref[Sampler]
Clone sampler. Maps to llama_sampler_clone.
func Sampler.free(s: ref[Sampler])
Free sampler resources. Maps to llama_sampler_free.
func sampler.reset()
Reset sampler state. Maps to llama_sampler_reset.
func sampler.sample(ctx: ref[Context], idx: Int): Token
Sample next token. Maps to llama_sampler_sample.
sampler.name: CharsPtr
Get sampler name. Maps to llama_sampler_name.
func sampler.accept(token: Token)
Accept token for tracking. Maps to llama_sampler_accept.
func sampler.apply(cand: ref[TokenDataArray])
Apply sampler to candidates. Maps to llama_sampler_apply.
func sampler.chainAdd(s: ref[Sampler])
Add sampler to chain. Maps to llama_sampler_chain_add.
func sampler.chainGet(idx: Int): ref[Sampler]
Get sampler from chain. Maps to llama_sampler_chain_get.
sampler.chainCount: Int
Get chain length. Maps to llama_sampler_chain_n.
func sampler.chainRemove(idx: Int): ref[Sampler]
Remove sampler from chain. Maps to llama_sampler_chain_remove.
sampler.perfSampler: PerfSamplerData
Get performance data. Maps to llama_perf_sampler.
func sampler.perfSamplerPrint()
Print performance data. Maps to llama_perf_sampler_print.
func sampler.perfSamplerReset()
Reset performance counters. Maps to llama_perf_sampler_reset.
Model vocabulary. Maps to llama_vocab.
func vocab.tokenToPiece(token: Token, buf: CharsPtr, length: Int, lstrip: Int, special: Bool): Int
Convert token to text. Maps to llama_token_to_piece.
vocab.eos: Token
Get end-of-sequence token. Maps to llama_vocab_eos.
func vocab.isEog(token: Token): Bool
Check if token is end-of-generation. Maps to llama_vocab_is_eog.
KV cache memory management. Maps to llama_memory.
func memory.clear(data: Bool)
Clear KV cache. Maps to llama_memory_clear.
func memory.seqRemove(seq: SeqId, p0: Pos, p1: Pos): Bool
Remove sequence range. Maps to llama_memory_seq_rm.
func memory.seqCopy(src: SeqId, dst: SeqId, p0: Pos, p1: Pos)
Copy sequence range. Maps to llama_memory_seq_cp.
func memory.seqKeep(seq: SeqId)
Keep only specified sequence. Maps to llama_memory_seq_keep.
func memory.seqAdd(seq: SeqId, p0: Pos, p1: Pos, delta: Pos)
Add position delta to sequence. Maps to llama_memory_seq_add.
func memory.seqDiv(seq: SeqId, p0: Pos, p1: Pos, d: Int)
Divide positions in sequence. Maps to llama_memory_seq_div.
LoRA adapter support. Maps to llama_adapter_lora.
func Adapter.loraInit(model: ref[Model], path: CharsPtr): ref[Adapter]
Load LoRA adapter. Maps to llama_adapter_lora_init.
func Adapter.loraFree(ad: ref[Adapter])
Free LoRA adapter. Maps to llama_adapter_lora_free.
func backendInit()
Initialize llama backend. Maps to llama_backend_init.
func backendFree()
Free llama backend. Maps to llama_backend_free.
func numaInit(strategy: Ggml.NumaStrategy)
Initialize NUMA. Maps to llama_numa_init.
func timeUs(): Int[64]
Get current time in microseconds. Maps to llama_time_us.
func maxDevices(): ArchWord
Get maximum number of devices. Maps to llama_max_devices.
func maxParallelSequences(): ArchWord
Get max parallel sequences. Maps to llama_max_parallel_sequences.
func supportsMmap(): Bool
Check mmap support. Maps to llama_supports_mmap.
func supportsMlock(): Bool
Check mlock support. Maps to llama_supports_mlock.
func supportsGpuOffload(): Bool
Check GPU offload support. Maps to llama_supports_gpu_offload.
func supportsRpc(): Bool
Check RPC support. Maps to llama_supports_rpc.
func printSystemInfo(): CharsPtr
Get system info string. Maps to llama_print_system_info.
func tokenize(
vocab: ref[Vocab], text: CharsPtr, textLen: Int, tokens: ref[array[Token]],
nTokensMax: Int, addSpecial: Bool, parseSpecial: Bool
): Int
Convert text to tokens. Returns the number of tokens written. Maps to llama_tokenize.
func detokenize(
vocab: ref[Vocab], tokens: ref[array[Token]], nTokens: Int, text: CharsPtr,
textLenMax: Int, removeSpecial: Bool, unparseSpecial: Bool
): Int
Convert tokens to text. Returns the number of characters written. Maps to llama_detokenize.
func chatApplyTemplate(
tmpl: CharsPtr, chat: ref[array[ChatMessage]], nMsg: ArchWord,
addAssistant: Bool, buf: CharsPtr, bufLen: Int
): Int
Apply chat template to messages. Pass 0 for tmpl to use the model's default template. Returns the number of characters written. Maps to llama_chat_apply_template.
func chatBuiltinTemplates(out: ref[CharsPtr], len: ArchWord): Int
Get built-in template names. Maps to llama_chat_builtin_templates.
func logSet(cb: ptr[function(level: Int, text: CharsPtr, userData: ptr)], userData: ptr)
Set logging callback. Maps to llama_log_set.
See Examples/completion.alusus for a complete text completion example.
See Examples/chat.alusus for a multi-turn chat example with chat templates.
import "Apm";
Apm.importPackage("Alusus/Llama@0.2");
use Llama;
// 1. Initialize backend
Ggml.Backend.cpuLoad();
// 2. Load model
def modelParams: Model.Params = Model.getDefaultParams();
modelParams.nGpuLayers = 0; // CPU only
def model: ref[Model](Model.load("model.gguf", modelParams));
// 3. Create context
def ctxParams: Context.Params = Context.getDefaultParams();
ctxParams.nCtx = 2048;
def ctx: ref[Context](Context.initFromModel(model, ctxParams));
// 4. Create sampler chain
def chainParams: Sampler.ChainParams;
chainParams.noPerf = true;
def sampler: ref[Sampler](Sampler.chainInit(chainParams));
sampler.chainAdd(Sampler.initPenalties(64, 1.1, 0.0, 0.0));
sampler.chainAdd(Sampler.initTopK(40));
sampler.chainAdd(Sampler.initTopP(0.9, 1));
sampler.chainAdd(Sampler.initDist(0));
// 5. Tokenize and decode prompt
def tokens: array[Token, 512];
def nTokens: Int = tokenize(model.vocab, "Hello world", 11, tokens, 512, true, true);
def batch: Batch = Batch.getOne(tokens, nTokens);
ctx.decode(batch);
// 6. Generate tokens
while true {
def id: Token = sampler.sample(ctx, -1);
if model.vocab.isEog(id) break;
// Convert token to text and print
def buf: array[Char, 64];
detokenize(model.vocab, id~ptr~cast[ref[array[Token]]], 1, buf~ptr, 64, 0, 0);
// ... print buf
// Decode next token
def nextBatch: Batch = Batch.getOne(id~ptr~cast[ref[array[Token]]], 1);
ctx.decode(nextBatch);
}
// 7. Cleanup
Sampler.free(sampler);
Context.free(ctx);
Model.free(model);
Copyright (c) 2023-2024 The ggml authors Copyright (c) 2026 Alusus Software Ltd. for the Alusus language bindings.
This library follows the same license as llama.cpp (MIT License).