This document provides details about the REST API endpoints exposed by the inference application. The API is built using Node.js and Express, acting as a proxy to the vLLM OpenAI-compatible inference service running on the same machine. The API proxy provides secure access, SSL termination, and consistent error handling.
- Development:
http://localhost:8080 - Production:
- IP-based:
http://<ec2-instance-ip>:8080 - Domain-based:
https://infer.<your-domain>(when DNS records and HTTPS are enabled)
- IP-based:
The API currently does not implement authentication. Future versions will include:
- API key authentication
- JWT token-based authentication
- AWS IAM-based authentication
Verify the API service is running properly.
GET /health{
"status": "ok",
"message": "API and vLLM service are healthy",
"version": "1.1.0"
}200 OK: Service is healthy503 Service Unavailable: vLLM service is not available
Get basic information about the API service.
GET /{
"message": "vLLM Inference API",
"timestamp": "2025-03-12T12:34:56.789Z",
"version": "1.1.0",
"modelInfo": "solidrust/Hermes-3-Llama-3.1-8B-AWQ"
}200 OK: Request successful
Get information about the available models.
GET /v1/models{
"object": "list",
"data": [
{
"id": "solidrust/Hermes-3-Llama-3.1-8B-AWQ",
"object": "model",
"created": 1710323150,
"owned_by": "vLLM"
}
]
}200 OK: Request successful503 Service Unavailable: vLLM service unavailable
Submit chat requests in OpenAI format for inference processing.
POST /v1/chat/completions{
"model": "<model-name>",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Tell me about AWS EC2." }
],
"max_tokens": 512,
"temperature": 0.7,
"stream": false
}{
"id": "cmpl-7a2e8938ebc44755bdcc45d37df1b106",
"object": "chat.completion",
"created": 1710323152,
"model": "<model-name>",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "AWS EC2 (Elastic Compute Cloud) is a web service provided by Amazon Web Services that offers resizable compute capacity in the cloud..."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 27,
"completion_tokens": 142,
"total_tokens": 169
}
}200 OK: Inference successful400 Bad Request: Invalid input data500 Internal Server Error: Processing error503 Service Unavailable: vLLM service unavailable
Submit text prompts for completion in OpenAI format.
POST /v1/completions{
"model": "<model-name>",
"prompt": "Write a short paragraph about AWS EC2",
"max_tokens": 256,
"temperature": 0.7
}{
"id": "cmpl-abc123def456",
"object": "text.completion",
"created": 1710325001,
"model": "<model-name>",
"choices": [
{
"text": "AWS EC2 (Elastic Compute Cloud) is Amazon's flagship cloud computing service that allows users to rent virtual computers...",
"index": 0,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 8,
"completion_tokens": 54,
"total_tokens": 62
}
}Convert text to tokens according to the model's tokenizer.
POST /tokenize{
"model": "<model-name>",
"messages": [
{ "role": "user", "content": "Hello, how are you?" }
]
}{
"model": "<model-name>",
"prompt": "Hello, how are you?"
}{
"tokens": [1, 23, 5, 7890, 12, 35, 23, 789]
}Convert token IDs back to text.
POST /detokenize{
"model": "<model-name>",
"tokens": [1, 23, 5, 7890, 12, 35, 23, 789]
}{
"text": "Hello, how are you?"
}Get the version information of the vLLM server and proxy.
GET /version{
"version": "0.2.5",
"proxy_version": "1.1.0"
}Generate embeddings for input text.
POST /v1/embeddings{
"model": "<embedding-model-name>",
"input": "Text to embed",
"encoding_format": "float"
}{
"object": "embedding",
"embedding": [0.1, 0.2, -0.3, 0.4, ...],
"model": "<embedding-model-name>"
}Rerank a list of documents according to their relevance to a query.
POST /rerank{
"model": "<rerank-model-name>",
"query": "How does AWS EC2 pricing work?",
"documents": [
"EC2 instances are billed by the hour with different rates for different instance types.",
"AWS provides different pricing models including On-Demand, Reserved, and Spot instances.",
"Cloud computing enables organizations to be more agile and reduce hardware costs."
],
"top_n": 2
}{
"model": "<rerank-model-name>",
"results": [
{
"document": "AWS provides different pricing models including On-Demand, Reserved, and Spot instances.",
"index": 1,
"relevance_score": 0.89
},
{
"document": "EC2 instances are billed by the hour with different rates for different instance types.",
"index": 0,
"relevance_score": 0.76
}
]
}Calculate the relevance score between two pieces of text.
POST /score{
"model": "<score-model-name>",
"text_1": "How does AWS EC2 pricing work?",
"text_2": "EC2 instances are billed by the hour with different rates for different instance types."
}{
"model": "<score-model-name>",
"score": 0.76
}All error responses follow this format:
{
"error": "Human-readable error message",
"details": "Additional error details",
"path": "/endpoint/path"
}Currently, the API does not implement rate limiting. Future versions will include rate limiting based on:
- IP address
- API key
- User/Client ID
The API follows semantic versioning (MAJOR.MINOR.PATCH). The current version can be found in the response from the root endpoint or the /health endpoint.
To run the API locally:
-
Navigate to the app directory:
cd app -
Install dependencies:
npm install
-
Start the server:
npm start
-
The server will be available at
http://localhost:8080
To build and run the API using Docker:
-
Build the image:
docker build -t inference-app . -
Run the container:
docker run -p 8080:8080 -e VLLM_HOST=host.docker.internal -e MODEL_ID="<your-model-id>" inference-app -
The server will be available at
http://localhost:8080
Examples of testing the API with cURL:
Health check:
curl http://localhost:8080/healthRoot endpoint:
curl http://localhost:8080/Models endpoint:
curl http://localhost:8080/v1/modelsChat completions:
curl -X POST \
http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "<your-model-id>",
"messages": [
{"role": "user", "content": "Explain what AWS EC2 is in one paragraph."}
],
"max_tokens": 100
}'To change the model used by the inference solution:
-
Edit the
terraform.tfvarsfile and modify themodel_idparameter:# vLLM Configuration model_id = "new-model/name-here"
-
Adjust
max_model_lenbased on the context window size of your chosen model. -
If needed, change the GPU instance type by modifying the
gpu_instance_typeparameter. -
Increment the
ec2_instance_versionto force a replacement of the EC2 instance:# Deployment version - increment to force replacement of EC2 instance ec2_instance_version = 3
-
Apply the changes with Terraform:
terraform apply
This will create a new EC2 instance with the updated model configuration, while maintaining the same API endpoints and functionality.
The API will be enhanced with the following features in future releases:
-
Authentication and Authorization:
- API key management
- OAuth 2.0 / OpenID Connect integration
- Role-based access control
-
Enhanced Inference Capabilities:
- Support for audio and vision models
- Function calling support
- Batched inference requests
- Asynchronous inference processing
-
Performance Features:
- Response caching
- Request queuing
- Throttling controls
-
Monitoring and Logging:
- Detailed request/response logging
- Performance metrics
- Tracing support
-
Documentation:
- OpenAPI (Swagger) specification
- Interactive documentation