GPT4All

How to configure and use it?

Pre-Requisite(s)

  • model (optional) - Set which model you want to use. Defaults to orca-mini-3b.ggmlv3.q4_0

Running in a Colab/Kaggle/Python scripts(s)

from genai_stack.model import Gpt4AllModel

llm = Gpt4AllModel.from_kwargs()
model_response = llm.predict("How many countries are there in the world?")
print(model_response["result"])

Running the model in a webserver

If you want to run the model in a webserver and interact with it with HTTP requests, the model provides a way to run it.

  1. As a Python script

from fastapi.responses import JSONResponse
from genai_stack.model import Gpt4AllModel

llm = Gpt4AllModel.from_kwargs()
llm.run_http_server(response_class=JSONResponse)

A server should start as below

Make HTTP requests. URL - http://localhost:8082/predict/

  1. As a CLI

Create a model.json file with the following contents:

Run the below command:

PreviousOpenAINextCustom Model

Last updated