GPT4All
How to configure and use it?
Pre-Requisite(s)
model
(optional) - Set which model you want to use. Defaults toorca-mini-3b.ggmlv3.q4_0
Running in a Colab/Kaggle/Python scripts(s)
from genai_stack.model import Gpt4AllModel
llm = Gpt4AllModel.from_kwargs()
model_response = llm.predict("How many countries are there in the world?")
print(model_response["result"])
Running the model in a webserver
If you want to run the model in a webserver and interact with it with HTTP requests, the model provides a way to run it.
As a Python script
from fastapi.responses import JSONResponse
from genai_stack.model import Gpt4AllModel
llm = Gpt4AllModel.from_kwargs()
llm.run_http_server(response_class=JSONResponse)
A server should start as below
INFO: Started server process [137717]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8082 (Press CTRL+C to quit)
Make HTTP requests. URL - http://localhost:8082/predict/
import requests
response = requests.post("http://localhost:8082/predict/", data="How many countries are there in the world?")
print(response.text)
As a CLI
Create a model.json
file with the following contents:
{
"model": {
"name": "gpt4all",
"fields": {
"model": "ggml-gpt4all-j-v1.3-groovy"
}
}
}
Run the below command:
genai-stack start --config_file model.json
โโโโโโโ โโโโโโโโโโโโ โโโ โโโโโโ โโโ โโโโโโโโโโโโโโโโโ โโโโโโ โโโโโโโโโโ โโโ
โโโโโโโโ โโโโโโโโโโโโโ โโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโ
โโโ โโโโโโโโโโ โโโโโโ โโโโโโโโโโโโโโ โโโโโโโโ โโโ โโโโโโโโโโโ โโโโโโโ
โโโ โโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโ โโโโโโโโ โโโ โโโโโโโโโโโ โโโโโโโ
โโโโโโโโโโโโโโโโโโโโ โโโโโโโโโ โโโโโโ โโโโโโโโ โโโ โโโ โโโโโโโโโโโโโโ โโโ
โโโโโโโ โโโโโโโโโโโ โโโโโโโโ โโโโโโ โโโโโโโโ โโโ โโโ โโโ โโโโโโโโโโ โโโ
INFO: Started server process [641734]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8082 (Press CTRL+C to quit)
Last updated