OpenAI’s New GPT-4o Interact Using Voice Or Video In The Same Model

The brand-new, free flagship “omnimodel” from the firm resembles an enhanced Siri or Alexa.

GPT-4o is a new type of AI model that OpenAI just unveiled. It allows for real-time text, video streaming from your phone, and voice conversations. According to the firm, the model is being rolled out over the next few weeks and will be available for free to all users via the GPT app and the online interface. More requests may be made by users that sign up for OpenAI’s subscription tiers, which begin at $20 per month.

The latest release was demonstrated live by OpenAI CTO Mira Murati on Tuesday, May 14, the day before Google is scheduled to reveal its own AI innovations at its annual I/O conference.

Related Article: With the release of GPTo, OpenAI enhances ChatGPT’s text, graphic, and audio features

Similar features were provided by GPT-4, allowing consumers to engage with OpenAI’s AI products in a variety of ways. However, it separated them into distinct models, which resulted in slower reaction times and perhaps more computational expenses. These features have now been combined into a single model, which Murati referred to as a “omnimodel,” by GPT-4o. According to her, this translates into quicker reactions and more seamless task transitions.

According to the company’s presentation, the end result is a conversational assistant that functions similarly to Siri or Alexa but is able to handle far more intricate requests.

Regarding the demonstration, Murati stated, “We’re looking at the future of interaction between ourselves and the machines.” “We believe that GPT-4o is really moving that paradigm toward a more natural kind of cooperation in the future.”

Researchers from OpenAI Barret Zoph and Mark Chen went over some uses for the new model. Its live chat capability was the most amazing feature. When you interrupted the model in the middle of its replies, it would pause, pay attention, and change direction.

Additionally, OpenAI demonstrated how to alter the model’s tone. Chen abruptly intervened to demand a more dramatic voice and requested the model to recite a bedtime story “about robotics and love.” The model became more and more dramatic until Murati insisted that it fast change to a genuine robot voice, at which point it was excellent. It was remarkable how easily the AI discussion flowed, even if there were a few expected brief pauses as the model decided what to say next.

In real time, the model is also capable of reasoning through visual difficulties. Zoph recorded himself using his phone while he wrote the algebraic equation (3x + 1 = 4) on paper, with GPT-4o following along. Instead of giving him answers, he told it to act like a teacher would and lead him.

The model stated in a kind voice, “The first stage is to get all the keywords with x on one side.” “So, what should we do with that extra one, in your opinion?”

According to Murati, GPT-4o will keep track of users’ interactions, so the model “now has a feeling of continuity throughout all your chats.” The capability to browse through the discussions you have with the model, seek up facts in real time, and live translation are further benefits.

As is typical with a live demo, there were some kinks and errors. Awkwardly, GPT-4o’s voice may suddenly enter the discourse. It seemed to make an observation about one of the speakers’ attire without being asked to. However, it bounced back well once the protesters pointed out its mistake to the model. It appears to be responsive and useful in a number of mediums that other models have not yet integrated as well.

Many of OpenAI’s more potent capabilities, such as picture and video reasoning, were previously restricted to paying customers. The public will be able to interact with them for the first time using GPT-4o, however it’s unclear how many times you may use the model before paying for it. Paying customers will “continue to have up to five times the capacity constraints of our free users,” according to OpenAI.

---