Should --rag be compatible with --network?

[BansheeHero]

Am I wrong for expecting ramalama to:

1. Create the same network for the whole stack?
2. Create names so the RAG images can see each other?
3. Print the 8081 host and not the 8080 in stdout?
   • Pretty sure this is a artifact of the model running 8080, but the proxy running 8081
   • I would appreciate docs or overview of the RAG project. Little bit more into how the stack works:
     I think I found it: https://deepwiki.com/containers/ramalama/9.3-rag-server-architecture

Steps taken:

0. I have created a small RAG image called "martin" without specifying anything else.

1. I created a restricted network: podman network create --internal restricted-net

2. I run the same command that works without specifying name and network: ramalama serve --name ramalama-restricted --network="restricted-net" --rag=martin unsloth/Qwen3.5-4B-GGUF:UD-Q4_K_XL

3. The container times out on creation:

   ec3c951ad1b30d3f325bb5db2459d4de623aa0e042b800adfdbe475771f483ea
   7b59c44bbc7a6e52546a56b9e0aaeccbfe8b173edca8a2cb0a3d161203c4c241
   INFO:     Started server process [2]
   INFO:     Waiting for application startup.
   ERROR:    Traceback (most recent call last):
     File "/opt/venv/lib64/python3.14/site-packages/starlette/routing.py", line 638, in lifespan
       async with self.lifespan_context(app) as maybe_state:
                  ~~~~~~~~~~~~~~~~~~~~~^^^^^
     File "/usr/lib64/python3.14/contextlib.py", line 214, in __aenter__
       return await anext(self.gen)
              ^^^^^^^^^^^^^^^^^^^^^
     File "/usr/bin/rag_framework", line 394, in lifespan
       await _wait_for_llm(args.model_host, args.model_port)
     File "/usr/bin/rag_framework", line 378, in _wait_for_llm
       raise TimeoutError(f"LLM server at {host}:{port} did not become ready after {total_timeout}s")
   TimeoutError: LLM server at host.containers.internal:8081 did not become ready after 120s
   
   ERROR:    Application startup failed. Exiting.
   

[thangldw]

Your expectation is reasonable, but the current RAG launch path is not compatible with a Podman --internal network.

_rag_args() copies the selected network to both containers, but it still hardcodes the model address seen by the RAG container to host.containers.internal and assigns the model a separate host port. An internal Podman network cannot reach that host gateway, so rag_framework waits for host.containers.internal:8081 and eventually times out. Merely placing both containers on the same network does not help because RamaLama is not using container-to-container DNS for this hop.

The ports are intentional:

• 8080 is the public RAG proxy endpoint.
• 8081 is the backend model endpoint selected for the RAG proxy to call.

Current workaround: omit --network, or use a non-internal custom network. Preserving isolation requires a code change: give the model container a stable name, place both containers on the same network, and set rag_args.model_host to that container name instead of host.containers.internal.

The relevant paths are _rag_args and the RAG command builder. I would classify this as a networking gap/bug rather than a command error.

[BansheeHero]

The ports are intentional:
8080 is the public RAG proxy endpoint.
8081 is the backend model endpoint selected for the RAG proxy to call.

I figured as much, so users shouldn't users want to open a llama.cpp web interface at 8081? (I believe the required prompt injection doesn't work when using it.)
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
Because that proxy does not show the web interface when using serve instead of run. .

The relevant paths are _rag_args and the RAG command builder. I would classify this as a networking gap/bug rather than a command error.

Should I raise an issue then? With the documentation and public presentations on Ramalama - I think supporting internal networks seem to part of the mission statement.

I encountered both of these while trying to make a python app that runs on internal network.

[olliewalsh]

@BansheeHero this is likely AI slop. @thangldw is not a contributor to this project.

[BansheeHero]

@BansheeHero this is likely AI slop. @thangldw is not a contributor to this project.

Apologies for falling for it.

[olliewalsh]

@bmahabirbu PTAL

@thangldw are you bot/AI-agent?

[olliewalsh]

Apologies if you really are a human. The sudden burst of activity on your github account in the past few weeks looks very bot-like.

I think we can assume that everyone has access to AI agents to inspect the code etc... so I'm not sure there is any point in adding the agent output here. It's much better to just wait for @bmahabirbu to respond as he implement the rag logic.

[bmahabirbu]

Its an interesting proposal to have all the containers be on the same network the simplest way would create a pod so I think its a simple scope request

Btw I still think @thangldw is a bot lol

[olliewalsh]

FWIW I'm already looking into using a private network for rag in order to land an unrelated change that would break the current networking.

[BansheeHero]

Its an interesting proposal to have all the containers be on the same network the simplest way would create a pod so I think its a simple scope request

Thank you, should I do something else here? Issue?
Running it in a pod would probably solve my second question too: #2898

I think long-term I will continue encountering images that I want to run on --internal and provide them with OpenAI endpoint.
Ramalama seems like the project for me.

[olliewalsh]

I think we would need to explicitly manage the networks vs using pods in order to support docker too