Data Retrieval and Manipulation with Bagel

Bagel provides APIs that enable secure data retrieval and manipulation. This tutorial demonstrates how to leverage these APIs to manage and analyze data while maintaining privacy and security.

Prerequisites - for JavaScript

To ensure that all the code in the tutorial is working properly, you can follow these steps:

  1. Make sure you have node >= v14 installed

  2. Install the Bagel JavaScript client, run the following command in your terminal: npm install bagelml

  3. Installing dependencies:

Also, install the following dependencies:

Axios : npm install axios Node-Fetch: npm install node-fetch Form data: npm install form-data uuid: npm install uuid Buffer: npm install buffer

  1. Create a CSV file containing product data. You can use any text editor or spreadsheet software like Microsoft Excel or Google Sheets to create this file. Save the file with the name product_catalog.csv in your project directory or any directory you prefer.

Note: You can also create a JavaScript code snippet to Generate Mock Data, here's the example we're using for this tutorial:

You will need to install the following packages for the Mock Data Generation: Fs: npm install fs Json2csv: npm install json2csv

import fs from 'fs'
import { Parser } from 'json2csv'

// Mock product data
const products = [
  { id: '001', name: 'Laptop', category: 'Electronics', price: 899.99 },
  { id: '002', name: 'Smartphone', category: 'Electronics', price: 499.99 },
  { id: '003', name: 'Tablet', category: 'Electronics', price: 299.99 },
  { id: '004', name: 'Headphones', category: 'Electronics', price: 149.99 },
  { id: '005', name: 'Camera', category: 'Electronics', price: 399.99 },
  { id: '006', name: 'Keyboard', category: 'Electronics', price: 49.99 },
  { id: '007', name: 'Mouse', category: 'Electronics', price: 19.99 },
  { id: '008', name: 'Speaker', category: 'Electronics', price: 99.99 },
  { id: '009', name: 'Television', category: 'Electronics', price: 1299.99 },
  { id: '010', name: 'Console', category: 'Electronics', price: 399.99 },
  { id: '011', name: 'Shirt', category: 'Clothing', price: 29.99 },
  { id: '012', name: 'Jeans', category: 'Clothing', price: 39.99 },
  { id: '013', name: 'Dress', category: 'Clothing', price: 49.99 },
  { id: '014', name: 'Sweater', category: 'Clothing', price: 59.99 },
  { id: '015', name: 'Jacket', category: 'Clothing', price: 79.99 },
  { id: '016', name: 'Hat', category: 'Accessories', price: 19.99 },
  { id: '017', name: 'Scarf', category: 'Accessories', price: 14.99 },
  { id: '018', name: 'Watch', category: 'Accessories', price: 199.99 },
  { id: '019', name: 'Sunglasses', category: 'Accessories', price: 99.99 },
  { id: '020', name: 'Backpack', category: 'Accessories', price: 49.99 },
]

// Define the fields for the CSV file
const fields = ['id', 'name', 'category', 'price']
const json2csvParser = new Parser({ fields })
const csv = json2csvParser.parse(products)

// Write the CSV data to a file
fs.writeFile('product_catalog.csv', csv, function (err) {
  if (err) throw err
  console.log('Mock product data generated and saved to product_catalog.csv')
})

Step 1: Create a VECTOR asset

Before we can begin managing our asset, we need to create a new vector asset within Bagel. This can be achieved using the POST /api/v1/asset endpoint:

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)

const api_key = "insert api key" // e.g "2347898767897ghjghjdsdnuisd8"
const payload = {
    dataset_type: "input dataset type", // e.g "RAW", "VECTOR", "MODEL"
    title: "Input title", // e.g "Apple"
    category: "inpute category", // e.g "Cat1", "Cat2"
    details: "input details", // e.g "testing"
    tags: ["Optional input"], // e.g ["AI", "DEMO", "TEST"]  or [ ]
    user_id: "input user id" // e.g "345281182308180743453"
}

// Calling the create asset method
const createAsset = async () => {
  // get response of created asset
  const res = await client.create_asset(payload, apiKey)
  console.log(res)
}

createAsset()

Step 2: Uploading Product Data

Upload the product catalog to your asset using a POST request (Vector Asset). This code uses the requests library to send a POST request to Bagel's API endpoint, providing the asset ID and the file data (product_catalog.csv). The response indicates whether the upload was successful:

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)

const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"
const file_path = "input file path" // e.g "product_catalog.csv" or "path/dir/file.csv"

// Calling the file upload method
const fileUpload = async () => {
    const res = await client.add_file(asset_id, file_path, api_key)
    console.log(res)
}

fileUpload()

Step 3: Add embeddings/Data to Vector Asset

Next, you can add embeddings to a vector asset. This involves sending a POST request with metadata, document details, and IDs in the payload. The function formats the URL with the asset ID, sends the request, and processes the response, console logging the result.

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)


const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"

const payload = {
  metadatas: [{"input metadata"}], // e.g [{ "source": "testing" }]
  documents: ["input documents"], // e.g ["Hi man"]
  ids: ["input id"] // e.g ["jkfbnjfd-t84urb54hurugb-uuybdiubviwd"] this is manually generated by you
}

// Calling the add data to asset method
const addEmbedding = async () => {
    const res = await client.add_data_to_asset(asset_id, payload, api_key)
    console.log(res)
}

addEmbedding()

Step 4: Retrieving Product Data/Downloading file

Now that our product data has been uploaded, we can retrieve it using the GET /api.bageldb.ai/api/v1/jobs/asset/{}/files/{} endpoint:

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)


const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"

const file_name = "input file name" // e.g train.txt"

// Calling the download file method
consgt fileDownload = async () => {
    const res = await client.download_model_file(asset_id, file_name, api_key)
    console.log(res)
}
fileDownload()

This code sends a GET request to the /api.bageldb.ai/api/v1/jobs/asset/{}/files/{} endpoint, providing the asset_id as a path parameter. The response contains the product data.

Step 5: Querying Assets

Bagel allows you to filter and retrieve specific subsets of data. This involves sending a POST request with a JSON payload containing the query parameters. The function formats the URL with the asset ID, sends the request, and processes the response, console logging the result or any error details.

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)


const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"


const payload = {
    where: {
        // category: "Cat2",
    },
    where_document: {
        // is_published: true,
    },
    // query_embeddings: [em],
    n_results: 1,
    include: ["metadatas", "documents", "distances"],
    query_texts: ["Badman"],
    padding: false,
}

// Calling the query asset method
const queryAsset = async () => {
    const res = await client.query_asset(asset_id, payload, api_key)
    console.log(res)
}

queryAsset()

Step 6: Updating Asset

You can update existing asset details by sending a PUT request. This code constructs the URL with the asset ID, sends the request with the updated payload, and console logs the response. If there's an error, it console logs the error details:

import { Settings, Client } from "bagelml"

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)


const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"

// Payload for updating the asset
const payload = {
   title: "input title", // e.g "Clothe"
   category: "input category", // e.g "Cat3",
   details: "input details", // e.g "Updated dataset description.",
   tags: ["input tags"] // e.g ["Updated", "Tags"] or [ ]
}

// Calling update asset method
const updateAsset = async () => {
    const res = await client.update_asset(asset_id, payload, api_key)
    console.log(res)
}

updateAsset()

By leveraging Bagel's APIs, developers can efficiently retrieve, query, and update data while ensuring data privacy and security.

Need extra help or just want to connect with other developers? Join our community on Discord for additional support 👾

PreviousBagel_JSNextFine-Tuning AI Models

Last updated