Multimodal Integration with OpenAI

Transcript

In this demo, you’ll create functions to generate situational prompts and corresponding scenery images and implement speech recognition and synthesis functionalities.

Start by defining a function to generate situational prompts. This function will create an initial situational context and a response.

Begin by writing the skeleton of your function.

# Function to generate a situational prompt for practicing English
def generate_situational_prompt(seed_prompt=""):
    # Define additional prompt instructions
    additional_prompt = """
    Then create an initial response to the person. If the situation
      is "ordering coffee in a cafe.", then the initial response will
      be, "Hello, what would you like to order?". Separate the initial
      situation and the initial response with a line containing "====".
      Something like:
        "You're ordering coffee in a cafe.
        ====
        'Hello, there. What would you like to order?'"
        Limit the output to 1 sentence.
    """

In this initial part of the function, you set up the additional instructions that will guide the generation of situational prompts. The additional_prompt variable provides a template for the type of response you expect.

Next, handle the input seed_prompt and construct the full prompt accordingly.

    # Check if a seed prompt is provided and create the seed
    # phrase accordingly
    if seed_prompt:
        seed_phrase = f"""Generate a second-person POV situation
          for practicing English with this seed prompt: {seed_prompt}.
        {additional_prompt}"""
    else:
        seed_phrase = f"""Generate a second-person POV situation
          for practicing English, like meeting your parents-in-law,
          etc.
        {additional_prompt}"""

Here, you check if you have a specific seed_prompt. If so, you incorporate it into your seed_phrase. Otherwise, use a general prompt for generating a situation.

Now, use GPT to generate your situational prompt.

    # Use GPT to generate a situation for practicing English
    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {"role": "system", "content": "You are a creative writer.
          Very very creative."},
        {"role": "user", "content": seed_phrase}
      ]
    )

In this segment, you call the GPT model to generate the situational prompt. You pass in the seed_phrase along with a role specification for the system and user.

Finally, extract and return the generated message from the response.

    # Extract and return the situation and the initial response
    # from the response
    message = response.choices[0].message.content

    # Return the generated message
    return message

Now, the function is complete. Test the function to ensure it’s working correctly.

# Test the function to generate a situational prompt
generate_situational_prompt()

Then, test it again with the seed prompt.

# Test the function to generate a situational prompt with a seed prompt
generate_situational_prompt("comics exhibition")

Next, create a function to generate a scenery image that matches the situational prompt. This function uses the DALL-E model.

# Generate an image based on the situational prompt

# Import necessary libraries for image processing and display
import requests
from PIL import Image
from io import BytesIO

def generate_situation_image(dalle_prompt):
    # Generate an image using the DALL-E 3 model with the provided prompt
    response = client.images.generate(
      model="dall-e-3", # Specify the model to use
      prompt=dalle_prompt, # The prompt describing the image to generate
      size="1024x1024", # Specify the size of the generated image
      n=1, # Number of images to generate
    )

    # Retrieve the URL of the generated image
    image_url = response.data[0].url

    # Download the image from the URL
    response = requests.get(image_url)

     # Open the image using PIL
    img = Image.open(BytesIO(response.content))

    # Return the image object
    return img

Then, create a function to display the image.

# Display the image in the cell
import matplotlib.pyplot as plt

# Display the image in the cell
def display_image(img):
    plt.imshow(img)
    plt.axis('off')
    plt.show()

Next, combine both functions to generate a situation and its matching image.

# Combine the functions to generate a situational prompt and
# its matching image
full_response = generate_situational_prompt("cafe")
initial_situation_prompt = full_response.split('====')[0].strip()
print(initial_situation_prompt)
img = generate_situation_image(initial_situation_prompt)
display_image(img)

At first, you get the situational prompt with the seed prompt “cafe”. But the situational prompt includes more than just a situation. It has the initial response from a person in that situation. In this example, the initial response could be a greeting from an employee at the cafe. But in generating an image representing the situation, you don’t need that initial response. So, you have to take it out first. The code, full_response.split('====')[0].strip(), splits the full response at the delimiter ==== and takes the first part (which is the initial situation prompt). The strip() method is used to remove any leading or trailing whitespace from the string.

Now, create a function to play the audio file. Add the following code to the Jupyter Lab:

# Play the audio file

# Import necessary libraries for audio processing and display
import librosa
from IPython.display import Audio, display

# Function to play a speech file
def play_speech(file_path):
    # Load the audio file using librosa
    y, sr = librosa.load(file_path)

    # Create an Audio object for playback
    audio = Audio(data=y, rate=sr, autoplay=True)

    # Display the audio player
    display(audio)

This function, play_speech, uses the librosa library to load an audio file from the provided file_path. It then creates an Audio object with the loaded data and sample rate, enabling playback. Finally, it uses the display function from IPython to show an audio player in the Jupyter Lab, allowing users to listen to the audio.

Next, create a function to generate speech from a text prompt using a text-to-speech (TTS) model.

# Function to generate speech from a text prompt
def speak_prompt(speech_prompt, autoplay=True,
  speech_file_path="speech.mp3"):
    # Generate speech from the grammar feedback using TTS
    response = client.audio.speech.create(
      model="tts-1",
      voice="alloy",
      input=speech_prompt
    )

    # Save the synthesized speech to the specified path
    response.stream_to_file(speech_file_path)

    # Sometimes you want to play the speech automatically,
    # sometimes you do not
    if autoplay:
        # Play the synthesized speech
        play_speech(speech_file_path)

This function, speak_prompt, uses a text-to-speech (TTS) model to generate speech from the provided speech_prompt. The generated speech is saved to a specified file path. If autoplay is set to True, the function will automatically play the synthesized speech using the play_speech function.

Play the initial response based on the situational prompt.

# Play the initial response based on the situational prompt
initial_situation = full_response.split('====')[1].strip()
speak_prompt(initial_situation)

Create a function to transcribe speech into text.

# Function to transcribe speech from an audio file
def transcript_speech(speech_filename="my_speech.wav"):
    with open(speech_filename, "rb") as audio_file:
        # Transcribe the audio file using the Whisper model
        transcription = client.audio.transcriptions.create(
          model="whisper-1",
          file=audio_file,
          response_format="json",
          language="en"
        )
    # Return the transcribed text
    return transcription.text

Transcribe the speech. Then, print the transcribed text.

# Transcribe the audio
transcripted_text = transcript_speech("audio/cappuccino.m4a")

# Print the transcribed text
print(transcripted_text)

Combine the initial response and transcribed text to create a conversation history.

# Function to create a conversation history
def creating_conversation_history(history, added_response):
    history = f"""{history}
====
'{added_response}'
"""
    return history

Now, use the function to create and print the conversation history.

# Create and print the conversation history
history = creating_conversation_history(full_response, transcripted_text)
print(history)

Generate a continuation of the conversation based on the history.

# Function to generate a conversation based on the conversation history
def generate_conversation_from_history(history):
    prompt = """Continue conversation from a person based on this
      conversation history and end it with '\n====\n'.
      Limit it to max 3 sentences.
      This is the history:"""
    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {"role": "system", "content": "You are a creative writer.
          Very very creative."},
        {"role": "user", "content": f"{prompt}\n{history}"}
      ]
    )
    # Extract and return the generated conversation
    message = response.choices[0].message.content
    return message

This function, generate_conversation_from_history, continues a conversation based on a given history. It constructs a prompt that instructs GPT to continue the conversation and limits the response to a maximum of three sentences. The generated response is then extracted and returned.

Generate and print the conversation based on the history.

# Generate and print the conversation based on the history
conversation = generate_conversation_from_history(history)
print(conversation)

Combine the conversation history with the new conversation and print it.

# Combine the conversation history with the new conversation
combined_history = history + "\n====\n" + conversation

# Print the combined history
print(combined_history)

Generate and display a scenery image based on the combined history.

# Generate a scenery image based on the combined history
dalle_prompt = "Generate a scenery based on this conversation: "
  + combined_history
img = generate_situation_image(dalle_prompt)

# Display the generated image
display_image(img)

Finally, generate and play the prompt based on the new conversation.

# Generate and play the prompt based on the new conversation
speak_prompt(conversation)

In this lesson’s next segment, you’ll learn how to build the UI with Gradio.