[Home Lab] AI Integration: LocalAI, Chroma, and Langchain4j

By See Hiong

December 29, 2023 - 3 minutes read - 544 words

Referring to the Building an AI application with Langchaing4j guide, the deployment of necessary Docker images, LocalAI, and Chroma to our Home Lab is outlined.

Creating custom LocalAI image

Begin with pulling the latest image using the easy docker setup guide:

docker pull quay.io/go-skynet/local-ai:v2.2.0

Now, run LocalAI from the ~/localai folder and download a model:

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:v2.2.0 --models-path /models --context-size 2000 --threads 4 --debug=true

# The model will be downloaded to ~/localai/models folder
curl http://127.0.0.1:8080/models/apply -H "Content-Type: application/json" -d '{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml"}'

homelab-localai-folder-structure

Next, create a Dockerfile and include the downloaded model under our custom image:

FROM quay.io/go-skynet/local-ai:v2.2.0

ENV HOST 0.0.0.0

RUN mkdir /models

COPY ./models /models

EXPOSE 8080

CMD ["/usr/bin/local-ai"]

Build it with:

docker build . -t localai:v2.2.0

# Test run the custom image
docker run -p 8080:8080 -ti --rm localai:v2.2.0 --models-path /models --context-size 2000 --threads 4 --debug=true

# Check the model
curl http://len:8080/models 

# Tag the image
docker tag localai:v2.2.0 192.168.68.115:30500/localai:v2.2.0

# Push the image (refer "Docker Image Push and Deployment" of Deploying OpenAI-Compatible LLAMA CPP Server in Home Lab with K3S)
docker push 192.168.68.115:30500/localai:v2.2.0

localai-custom-image-with-gpt4all-model

Deploying Custom LocalAI to Home Lab

This is the localai-deploy.yaml file targeting to run from the len server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: localai-server
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: localai-server
  template:
    metadata:
      labels:
        app: localai-server
    spec:
      containers:
        - name: localai-server
          image: 192.168.68.115:30500/localai:v2.2.0
          ports:
            - containerPort: 8080
          args:
            - "--models-path"
            - "/models"
      nodeSelector:
        kubernetes.io/hostname: len

This is the localai-svc.yaml file, exposing nodePort as 30808:

apiVersion: v1
kind: Service
metadata:
  name: localai-server-svc
  namespace: llm
spec:
  selector:
    app: localai-server
  type: NodePort
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
      nodePort: 30808

Deploy to the Home Lab:

kca localai-deploy.yaml
kca localai-svc.yaml

localai-running-in-home-lab

NOTE: If you are unable to pull the image from the agent, please refer to my previous post and perform the following:

sudo vi /etc/systemd/system/k3s-agent.service

Add the following at the bottom of the existing file:

ExecStart=/usr/local/bin/k3s \
    agent \
    --docker
    --insecure-registry=http://192.168.68.115:30500

Restart the k3s-agent:

sudo systemctl daemon-reload
sudo systemctl restart k3s-agent

Deploying Chroma to Home Lab

The HostPath /docker-data/chroma will be created in the HostName len. This is the chroma-deploy.yaml file to be deployed to our Home Lab:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chroma-server
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: chroma-server
  template:
    metadata:
      labels:
        app: chroma-server
        name: chroma-server
    spec:
      containers:
      - name: chroma-server
        image: ghcr.io/chroma-core/chroma:0.4.21
        ports:
        - containerPort: 8000
        volumeMounts:
        - name: chroma-data
          mountPath: /chroma/chroma
      volumes:
      - name: chroma-data
        hostPath:
          path: "/docker-data/chroma"
      nodeSelector:
        kubernetes.io/hostname: len

To expose this service for our usage, this is the chroma-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: chroma-server-svc
  namespace: llm
spec:
  selector:
    app: chroma-server
  type: NodePort
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8000
      nodePort: 30800

As usual, we will deploy using our alias command:

kca chroma-deploy.yaml
kca chroma-svc.yaml

chroma-running-in-home-lab

Integrating Langchain4j Application with Home Lab

Returning to the Langchain4j application codebase, certain changes are required to integrate our application into the Home Lab.

Embed Endpoint Modification

With the above setup, this is the URL change (http://len:30800) made to EmbeddingServer.java:

private EmbeddingStore<TextSegment> getEmbeddingStore() {
    if (embeddingStore == null) {
        embeddingStore = ChromaEmbeddingStore.builder().baseUrl("http://len:30800")
                .collectionName(randomUUID()).build();
    }
    return embeddingStore;
}

integrate-embed-endpoint-with-home-lab

Retrieve Endpoint Modification

Based on our LocalAI setup, in the ModelService.java, the LOCAL_AI_URL is changed to:

private static final String LOCAL_AI_URL = "http://len:30808";

integrate-retrieve-endpoint-with-home-lab

localai-home-lab-integrate-with-langchain4j-application