Skip to content

fix(server): keep LlamaProxy reusable after freeing its model - #2421

Open
cloudgpuapp wants to merge 1 commit into
abetlen:mainfrom
cloudgpuapp:fix/llama-proxy-reuse-r21
Open

cloudgpuapp wants to merge 1 commit into
abetlen:mainfrom
cloudgpuapp:fix/llama-proxy-reuse-r21

Conversation

@cloudgpuapp

Copy link
Copy Markdown

Calling LlamaProxy.free() removes _current_model entirely. Calling free() again, or asking the same proxy to load a model afterward, then raises AttributeError before reaching the existing empty-model handling.

Keep the attribute set to None after a successful close. A failed close still propagates its exception and retains the reference for a retry. This changes neither model selection nor the native close implementation and adds no concurrency guarantee.

Validation on macOS arm64, Python 3.12.16:

  • Five regression cases: repeated free, default reload, different-model reload, empty proxy, and failed-close retry. Original source: 4 failed / 1 passed. Patched: 5 passed.
  • The local run imported the real server/model.py and server/settings.py, mocked model creation, and replaced the native package at import. It did not load the compiled backend or a model. The test file itself is a normal pytest file using the package imports.
  • Full make lint and git diff --check passed.
  • Native/model-dependent make test, HTTP serving and GPU memory behavior were not run. In a built checkout with server/test extras, run python -m pytest tests/test_server_model_lifecycle.py -q for a normal-import check.

AI assistance was used to prepare the change and tests. This is a server-proxy lifecycle fix; it does not add a provider or promotional link.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant