How to deploy Chroma DB to AWS EC2 instance

Let's start with the already prepared AWS CloudFormation template, that will create an EC2 instance.

Usually, I start by using t3.small (vCPU: 2, Memory: 2GiB). Deployment will be performed in the existing Default VPC and using one, random AZ.

Once the EC2 instance is Initialized connect to it using SSH.

1. Let us first check if there are some updates that we can perform before we proceed to the installation of the needed tools and services.

$ sudo yum update
Last metadata expiration check: 0:03:24 ago on Wed Sep 27 19:00:36 2023.
Dependencies resolved.
Nothing to do.
Complete!

1. Let's start by installing Docker

    $ sudo yum install -y docker
   

2. Add group membership for the default ec2-user so you can run all docker commands without using the sudo command

    $ sudo usermod -a -G docker ec2-user
    $ id ec2-user
    uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user),4(adm),10(wheel),190(systemd-journal),992(docker)
    $ newgrp docker
   

3. Check if Docker is installed

    $ docker version
    Client:
     Version:           24.0.5
     API version:       1.43
     Go version:        go1.20.7
     Git commit:        ced0996
     Built:             Thu Aug 31 00:00:00 2023
     OS/Arch:           linux/amd64
     Context:           default
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
   

   As we can see, at the moment when I'm writing this document, the Docker version is 24.0.5. And furthermore Docker has not been started as a service.

4. Enable docker service at AMI boot time

    $ sudo systemctl enable docker.service
   
    Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
   

5. Start the Docker service

    $ sudo systemctl start docker.service
   

How to control the Docker service?

Use the systemctl command as follows:

$ sudo systemctl start docker.service #<-- start the service
$ sudo systemctl stop docker.service #<-- stop the service
$ sudo systemctl restart docker.service #<-- restart the service
$ sudo systemctl status docker.service #<-- get the service status

How to install docker-compose?

wget docker-compose for your version of OS

 $ wget https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)

$ sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/local/bin/docker-compose

$ sudo chmod -v +x /usr/local/bin/docker-compose
mode of '/usr/local/bin/docker-compose' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x)

Finally, let's check the version of docker-compose

$ docker-compose version
Docker Compose version v2.22.0

Run ChromaDB

Let's first create docker-compose.yaml file.

version: '3.9'

services:
  chroma:
    image: ghcr.io/chroma-core/chroma:latest
    container_name: chroma
    ports:
      - 8000:8000
    volumes:
      - "chroma-data:/chroma/.chroma/index"
    restart: always
    networks:
      - net

networks:
  net:
    driver: bridge

volumes:
  chroma-data:
    driver: local

Save and exit. You can start chroma-db service with following command:

$ docker-compose -f docker-compose.yaml -d
[+] Building 0.0s (0/0)                                                                                                                                                                                                               docker:default
[+] Running 1/1
 ✔ Container chroma  Started

If chroma-db service starts OK, you can check docker status by issuing command

$ docker ps -a
CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS          PORTS                                       NAMES
0351057e0fa8   ghcr.io/chroma-core/chroma:latest   "/docker_entrypoint.…"   26 seconds ago   Up 25 seconds   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp   chroma

And also you can check the ChromaDB docker logs

$ docker logs --follow 0351057e0fa8
Rebuilding hnsw to ensure architecture compatibility
Collecting chroma-hnswlib
  Downloading chroma_hnswlib-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 43.7 MB/s eta 0:00:00
Collecting numpy
  Downloading numpy-1.26.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 155.5 MB/s eta 0:00:00
Installing collected packages: numpy, chroma-hnswlib
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.4
    Uninstalling numpy-1.22.4:
      Successfully uninstalled numpy-1.22.4
  Attempting uninstall: chroma-hnswlib
    Found existing installation: chroma-hnswlib 0.7.3
    Uninstalling chroma-hnswlib-0.7.3:
      Successfully uninstalled chroma-hnswlib-0.7.3
Successfully installed chroma-hnswlib-0.7.3 numpy-1.26.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
INFO:     [27-09-2023 19:37:32] Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
DEBUG:    [27-09-2023 19:37:32] Starting component System
DEBUG:    [27-09-2023 19:37:32] Starting component Posthog
DEBUG:    [27-09-2023 19:37:32] Starting component SqliteDB
DEBUG:    [27-09-2023 19:37:32] Starting component LocalSegmentManager
DEBUG:    [27-09-2023 19:37:32] Starting component SegmentAPI
INFO:     [27-09-2023 19:37:32] Started server process [12]
INFO:     [27-09-2023 19:37:32] Waiting for application startup.
INFO:     [27-09-2023 19:37:32] Application startup complete.
INFO:     [27-09-2023 19:37:32] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

As we can see ChromaDB is ready to accept connections on port :8000.

NOTE: Please make sure, before you start using REST API to add the port :8000 to SecurityGroup which is protecting this EC2 instance as a new Ingress (inbound) rule.

To check if ChromaDB is ready to accept connections and future vectors and embeddings, let's do a simple GET to verify that it is working as expected:

$ curl http://[EC2-INSTANCE-PUBLIC-IP]:8000/api/v1/
{"nanosecond heartbeat":1695843951657666845}

As we can see everything is OK.