A Hands-on Exploration of the Azure OpenAI API (Part 4 of 6)

1. Retrieval-Augmented Generation

Let’s continue our journey and learn more about enhancing our model output using Retrieval-Augmented Generation (RAG).

As mentioned in Part 1, Chapter 4.3 of this workshop: Retrieval Augmented Generation (RAG), rather than solely depending on the default model’s existing knowledge, we can supplement the model with our data and boost the overall comprehension without re-training.

The two Jupyter Notebooks we are going to work with are:

1. P4-azure-openai-assistant-rag-data-preprocessing.ipynb
2. P4-azure-openai-rag.ipynb

In Notebook P4-azure-openai-assistant-rag-data-preprocessing.ipynb, we will conduct data pre-processing steps using the Azure OpenAI Assistant API using the GPT-4o-mini model version 2024–07–18. We will input data-wrangling instructions in text format and have the code interpreter of the Azure OpenAI Assistant execute our requirements.

In Notebook P4-azure-openai-rag.ipynb, we will implement RAG using ChromaDB as our Vector Database together with LangChain and the Azure OpenAI embedding model text-embedding-3-large. Subsequently, we will use the GPT-4o-mini model with the aid of RAG to create well-thought-out and flavourful…