Guide: Running on premises

We provide docker images to run our models on your infrastructure. You get all features of http://api.reka.ai but fully locally. No data ever leaves your servers. In fact, once you have downloaded the docker images and model weights, no internet connection is required.

This guide will explain how to run our full stack. We assume that you have access to a machine with at least 2 GPUs and that you have been provided credentials by Reka.

Note

You must first install the NVIDIA Container Toolkit on your machine.

Enter the credentials provided by Reka to download models and docker images from AWS:

aws configure

Download model weights and docker code:

volume=/tmp/checkpoints/draco-v1/
mkdir -p "$volume"
aws s3 cp s3://reka-models/draco "$volume" --recursive
aws s3 cp s3://reka-code/docker-compose.yml reka.yml 

Login to Docker (replace with the correct registry name)

aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin *********.dkr.ecr.eu-west-2.amazonaws.com

Download and start Reka containers:

docker-compose -f reka.yml up --detach

The model takes a couple of minutes to load. Once the server has successfully started, an "OK" message will be returned by http://localhost:8080/healthz.

To test the local model with retrieval, you can install the Python API and download a toy dataset:

pip install reka-api
aws s3 cp s3://reka-code/amazon_legal_docs.zip

You can use the Python API to ingest the new dataset locally and query the retrieval augmented model:

import reka
reka._SERVER="http://localhost:8000"
reka.API_KEY="on_prem_key"
reka.add_dataset("amazon_legal_docs.zip", name="amazon_legal_docs")
job_id = reka.prepare_retrieval(dataset_name="amazon_legal_docs")
# Should take about 30s.
while not reka.retrieval_job_status(job_id).is_done():
  time.sleep(5)
reka.chat('What is the refund policy for Amazon Prime memberships that were redeemed through a promotional or gift code?', retrieval_dataset="amazon_legal_docs")

The output should be something like:

Amazon Prime memberships that were redeemed through a promotional or gift code are not refundable. This means that if you purchased an Amazon Prime membership using a promotional code or gift card, you cannot return it or request a refund. The specific terms and conditions for refunds and returns of Prime memberships can vary depending on the country you are located in, so it is always a good idea to review the specific return policy for the country in which you are located.