From dbd99e7b4b1db6f62f313c16ec672d0e09ae1077 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?C=C3=A9sar=20Garc=C3=ADa?= Date: Sat, 11 Nov 2023 09:23:46 +0100 Subject: [PATCH] Update description.md (#1107) Added a section on how to customize low level args, proposing people to stick to suggested models. --- docs/description.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/description.md b/docs/description.md index d5e2e88..2b98247 100644 --- a/docs/description.md +++ b/docs/description.md @@ -174,6 +174,10 @@ is used. > It's highly encouraged that you fully read llama-cpp and llama-cpp-python documentation relevant to your platform. > Running into installation issues is very likely, and you'll need to troubleshoot them yourself. +#### Customizing low level parameters + +Currently not all the parameters of llama-cpp and llama-cpp-python are available at PrivateGPT's `settings.yaml` file. In case you need to customize parameters such as the number of layers loaded into the GPU, you might change these at the `llm_component.py` file under the `private_gpt/components/llm/llm_component.py`. If you are getting an out of memory error, you might also try a smaller model or stick to the proposed recommended models, instead of custom tuning the parameters. + #### OSX GPU support You will need to build [llama.cpp](https://github.com/ggerganov/llama.cpp) with