Skip to main content

Example: Prompt Engineering with ChatGPT and RAW

This example shows how to use RAW and ChatGPT to process survey data to read and classify customer complaints.

note

We want to give credit to Mr Sung Kim as this example is heavily inspired by his blog post, "Prompt Engineering with OpenAI GPT-3 API: A Real-World Example".

info

If you are not familiar with RAW we recommend checking out our Getting started guide first. To use RAW, you need an account which you can create and use for free here.

What is Prompt Engineering?​

Prompt engineering is the process of designing and refining the prompts or instructions given to AI language models, such as OpenAI's GPT-3, to improve their performance on specific tasks. The prompts are essentially the input given to the model that guides it to generate the desired output. Prompt engineering involves crafting the prompts in a way that makes the task at hand clear to the model and leads it to produce accurate and relevant results.

National Highway Traffic Safety Administration (NHTSA) Datasets​

In this example, we use a public dataset provided by the National Highway Traffic Safety Administration (NHTSA), which can be found on their Datasets and APIs page. Specifically, we will be using the customer complaint data for the 2020 Kia Telluride SUV.

We will interact with the dataset through the API interface. The data output will be in JSON format, which you can read using Json.InferAndRead:

Json.InferAndRead(
    "https://api.nhtsa.gov/complaints/complaintsByVehicle?make=kia&model=telluride&modelYear=2020"
)
note

The choice of the 2020 Kia Telluride SUV model is arbitrary. We are not affiliated with Kia Motors or any related entities. In fact, Kia has a good safety rating. The purpose of this example is solely to illustrate the concept of prompt engineering in natural language processing.

Using ChatGPT to Classify NHTSA Complaints​

Accessing the OpenAI API​

To use the OpenAI API, the first step is to obtain an API key for OpenAI. If you do not have one yet, you need to create an account and go to https://platform.openai.com/account/api-keys to generate a new API key.

Then we can access the completions API with the following Snapi code:

Json.InferAndRead(
    Http.Post(
        "https://api.openai.com/v1/chat/completions",
        token = Environment.Secret("openai-api-key"),
        headers = [{ "Content-Type", "application/json"}],
        bodyString = Json.Print({
            model: "gpt-3.5-turbo",
            messages: [{role: "user", content: "Hello!"}]
        })
    )
)

To learn more about the OpenAI REST API, refer to the documentation.

note

We stored the API key in a secret called openai-api-key. Instructions on how to create secrets can be found in here.

Providing Prompt Instructions​

Our aim is to categorize NHTSA complaints based on the relevant component involved. This can be achieved by providing ChatGPT with a prompt instruction that gives context to the model. By using the appropriate prompt message, the model can perform better and more accurately align with the task at hand. To learn more about this, refer to the guide "Best Practices for Prompt Engineering with OpenAI API".

This is what our prompt message looks like:

let
    promptInstruction = """Instruction: The text is from NHTSA consumer complaints for 2020 KIA TELLURIDE.
      Categorize the text to one or more available values. The available values are:
        - AIR BAGS
        - ELECTRICAL SYSTEM
        - ENGINE
        - EXTERIOR LIGHTING
        - FORWARD COLLISION AVOIDANCE
        - FUEL/PROPULSION SYSTEM
        - LANE DEPARTURE
        - LATCHES/LOCKS/LINKAGES
        - PARKING BRAKE
        - POWER TRAIN
        - SEATS
        - SERVICE BRAKES
        - SERVICE BRAKES, HYDRAULIC
        - STEERING
        - STRUCTURE
        - SUSPENSION
        - TRAILER HITCHES
        - UNKNOWN OR OTHER
        - VEHICLE SPEED CONTROL
        - VISIBILITY
        - VISIBILITY/WIPER"""

Final code​

In the following code we:

  • Create a function to get completion data from the OpenAI API.
  • Get the NHTSA complaints for the Kia Telluride model.
  • Provide the prompt instruction, along with the summary from the complaint, to the "gpt-3.5-turbo" model.
  • Collect the choices from the model together with the summary and component from the original complaint to be able to compare results.
// Function to get completion data from OpenAI REST interface.
chatCompletion(
    model: string,
    messages: list(record(role: string, content: string))
) =
    let
        tipe = type record(
            id: string,
            object: string,
            created: long,
            choices: list(
                record(
                    index: int,
                    message: record(role: string, content: string),
                    finish_reason: string
                )
            ),
            usage: record(
                prompt_tokens: int,
                completion_tokens: int,
                total_tokens: int
            )
        )
    in
        Json.Read(
            Http.Post(
                "https://api.openai.com/v1/chat/completions",
                token = Environment.Secret("openai-api-key"),
                headers = [{"Content-Type", "application/json"}],
                bodyString = Json.Print({model: model, messages: messages})
            ),
            tipe
        )


let
    // We will read consumer complaint data from the National Highway Traffic Safety Administration (NHTSA)
    // and use open AI chat GPT 3.5 to read the summary and guess the classification
    // We are getting only data for the model Kia Telluride (we could choose another model/manufacturer)
    nhtsa_complaints = Json.InferAndRead(
        "https://api.nhtsa.gov/complaints/complaintsByVehicle?make=kia&model=telluride&modelYear=2020"
    ).results,
    promptInstruction = """Instruction: The text is from NHTSA consumer complaints for 2020 KIA TELLURIDE. Categorize the text to one or more available values. The available values are:
        - AIR BAGS
        - ELECTRICAL SYSTEM
        - ENGINE
        - EXTERIOR LIGHTING
        - FORWARD COLLISION AVOIDANCE
        - FUEL/PROPULSION SYSTEM
        - LANE DEPARTURE
        - LATCHES/LOCKS/LINKAGES
        - PARKING BRAKE
        - POWER TRAIN
        - SEATS
        - SERVICE BRAKES
        - SERVICE BRAKES, HYDRAULIC
        - STEERING
        - STRUCTURE
        - SUSPENSION
        - TRAILER HITCHES
        - UNKNOWN OR OTHER
        - VEHICLE SPEED CONTROL
        - VISIBILITY
        - VISIBILITY/WIPER""",
    sample = Collection.Take(nhtsa_complaints, 3)
in
    Collection.Transform(
        sample,
        (x) ->
            {
                components: x.components,
                summary: x.summary,
                result: chatCompletion(
                    "gpt-3.5-turbo",
                    [
                        {role: "system", content: promptInstruction},
                        {role: "user", content: x.summary}
                    ]
                ).choices
            }
    )

This is the output:

[
  {
    "components": "EXTERIOR LIGHTING",
    "summary": "High beams stopped working.  No fault of owner.  Took to two Kia dealerships that would not pay the $1800 to replace lights.  We can not pass state inspection and dangerous driving dark roads at nights.  Vehicle still under “bumper to bumper warranty”.   Should be a recall or repaired by Kia.",
    "result": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "EXTERIOR LIGHTING."
        },
        "finish_reason": "stop"
      }
    ]
  },
  {
    "components": "ENGINE",
    "summary": "We experienced an issue of the engine stalling while my wife was stopping at a stop light on multiple occasions. I did not think to check the oil as the vehicle only had about 60k miles at the time. After taking it to KIA  here in Jacksonville NC they told us our oil was low and adivsed us to choose the most expensive oil change, i obliged with obvious reservations. After the oil change was completed they wanted it back right at 1,000 miles later to check the oil level. When we brought it back in they said it is within normal limits. I find it hard to believe that an engine should be burning oil this soon when we have kept up with routine maintenance by the dealership since we purchased the vehicle. I am now at a spot where I am adding checking/adding oil until we can get rid of this vehicle since KIA does not want to stand behing their product.",
    "result": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "- ENGINE \n- FUEL/PROPULSION SYSTEM"
        },
        "finish_reason": "stop"
      }
    ]
  },
  {
    "components": "EXTERIOR LIGHTING",
    "summary": "While driving all of a sudden high beams shut off and would not come back on. Attempted to fix issue by replacing bulbs. This did not fix the issue. Technician looked at the wiring into headlights and no issue was found. Have not been able to fix thus issue. It is going on a year now without having high beams. During my research multiple other owners have Same issue. I spoke to kia and was told 10 months ago that they are working on a recall for this issue. Still no recall issued. I have spoken to multiple other owners that have same make and model they have Same issue and had to pay out of pocket to replace both headlight assemblies to fix the issue. This is unacceptable and a high safety issue!",
    "result": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "EXTERIOR LIGHTING"
        },
        "finish_reason": "stop"
      }
    ]
  }
]

As we can see, OpenAI's model "gpt-3.5-turbo" was able to correctly classify the relevant component from the complaint text provided by the customer in most cases. According to the blog post by Sum King, the model achieved a matching accuracy of 95.23%."