Advanced Chat Completion Techniques - Instruction

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Fine-Tuning your Response

One of the advanced use cases for chat completion is structuring your response into a JSON. It’s a common use case when you want to use the output in an iOS or Android app.

Deterministic Output

One problem you might face in implementing this is that the output might say true one time and false one time for the same statement and source. You want to make this as deterministic as possible. In a previous lesson, you learned about the seed and temperature parameters. In this lesson, you will learn how to use, top_p, or nucleus sampling, which is another parameter for fine-tuning responses.

First Attempt

In Jupyter Lab, navigate to 04-advanced-chat-completion/Starter/lesson4.ipynb. You should see some code already there. Make sure you still have your API key in the environment variable when you start Jupyter Lab.

# 1
user_statement = "White sand is made from fish poop https://oceanservice.noaa.gov/facts/sand.html"

messages = [
 {"role": "user", "content": user_statement}
]

# 2
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages
)
print(response.choices[0].message.content)

The statement that "white sand is made from fish poop" is a simplification of a more complex process. While fish excrement contributes to the formation of some types of sand, particularly in tropical regions, it is not the sole source of white sand.

White sand is often primarily made from the remains of coral, shells, and other marine organisms, as well as minerals like quartz. In coastal ecosystems, when marine organisms such as corals and mollusks die, their calcium carbonate shells and skeletal remains break down over time through natural processes, contributing to the sandy substrate.

Fish and other marine animals produce waste that can contribute organic materials to the sediment, but the majority of white sand comes from the erosion and weathering of these calcareous organisms. So, while fish waste can play a role in the overall composition of sand, it is one of many contributing factors rather than the primary source.

JSON Output

The first thing to note about your previous output was that it was purely unstructured text. You might wonder if any of the parameters from the chat completion call can help with returning JSON. And you’re exactly right. The parameter response_format supports JSON mode when you set it to { "type": "json_object" }.

...
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages,
  response_format={ "type": "json_object" }
)
...

'messages' must contain the word 'json' in some form

System Prompt

Enter system prompts. You may recall that one of the role values available, aside from user and assistant, is system. You should be able to add the word json to the system prompt.

{
  "isFactTrue": <true or false>
  "explanation": <explanation>
}

# 1
user_statement = "White sand is made from fish poop https://oceanservice.noaa.gov/facts/sand.html"


# 2
SYSTEM_PROMPT = (
  "You are a fact checker. Verify the validity of the sentence provided by the user, given a reference. You must return a response in JSON format:"
  "{'isFactTrue': <true or false>, 'explanation': <explanation> }"
)
messages = [
 {"role": "system", "content": SYSTEM_PROMPT},
 {"role": "user", "content": user_statement}
]

# 3
response = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=messages,
 response_format={ "type": "json_object" }
)
print(response.choices[0].message.content)

{
  "isFactTrue": false,
  "explanation": "White sand is primarily composed of tiny fragments of coral, shells, and calcium carbonate, rather than being made from fish poop. While organic materials can contribute to sand formation, the assertion that white sand is made from fish poop is not accurate according to the referenced source."
}

Tool Calls

To fully use the reference URL, first your code has to notice the URL, and then decide that it needs to perform a web request in order to fetch the contents and pass the URL to that request.

Defining a Tool

To define a tool, you should first define a Python function. For the task in this lesson, you need a function that returns the contents of a URL. Add a new cell and write the following:

def get_text_from_url(url):
  if url == "https://oceanservice.noaa.gov/facts/sand.html":
    return '''The famous white-sand beaches of Hawaii, for example, actually come from the poop of parrotfish. The fish bite and scrape algae off of rocks and dead corals with their parrot-like beaks, grind up the inedible calcium-carbonate reef material (made mostly of coral skeletons) in their guts, and then excrete it as sand.'''

tools = [
  {
    # 1
    "type": "function",
    # 2
    "function": {
      # 3
      "name": "get_text_from_url",
      # 4
      "description": "Get the text contents of a url",
      # 5
      "parameters": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string",
            "description": "The url",
          },
        },
        "required": ["url"],
      },
    },
  }
]

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=messages,
  tools=tools,
  response_format={"type": "json_object"},
)
print(response.choices[0].message.content)

print(response)

ChatCompletion(
  choices=[
    Choice(
      finish_reason='tool_calls',
      index=0,
      logprobs=None,
      message=ChatCompletionMessage(
        content=None,
        role='assistant',
        function_call=None,
        tool_calls=[
          ChatCompletionMessageToolCall(
            id='call_m5CgudT097SnO6no43hoFmeb',
            function=Function(
              arguments='{"url":"https://oceanservice.noaa.gov/facts/sand.html"}',
              name='get_text_from_url'
            ),
            type='function'
          )
        ],
        refusal=None
      )
    )
  ],
  ...
)

response_message = response.choices[0].message
tool_calls = response_message.tool_calls
print(tool_calls)

[
  ChatCompletionMessageToolCall(
    id='call_m5CgudT097SnO6no43hoFmeb',
    function=Function(
      arguments='{"url":"https://oceanservice.noaa.gov/facts/sand.html"}',
      name='get_text_from_url'
    ),
    type='function'
  )
]

# 1
import json

if tool_calls:
  # 2
  available_functions = {
    "get_text_from_url": get_text_from_url,
  }
  # 3
  messages.append(response_message)
  # 4
  for tool_call in tool_calls:
    # 5
    function_name = tool_call.function.name
    function_to_call = available_functions[function_name]

    # 6
    function_args = json.loads(tool_call.function.arguments)

    # 7
    function_response = function_to_call(
      url=function_args.get("url"),
    )
    # 8
    messages.append(
      {
        "tool_call_id": tool_call.id,
        "role": "tool",
        "name": function_name,
        "content": function_response,
      }
    )

  # 9
  second_response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    response_format={"type": "json_object"},
  )

  # 10
  print(second_response.choices[0].message.content)

{
  "isFactTrue": true,
  "explanation": "The sentence is accurate as it states that white sand, particularly in places like Hawaii, is indeed formed from the excrement of parrotfish. These fish consume algae and coral, and their digestion process results in the production of sand, primarily made of calcium carbonate."
}
See forum comments
Download course materials from Github
Previous: Advanced Chat Completion Techniques - Introduction Next: Advanced Chat Completion Techniques - Demo