🦾How You Can Leverage:

*Leeeeeeeeeet’s get ready to LLM ruuuuuuuuuumble.

It’s been a while since we pitted top LLMs head to head in an Everyday AI episode. 

So, ya’ll told us you wanted a throw-down, so we threw it down live. 

// screenshot from Claude poll last week // 

With Anthropic’s new Claude 3.5 Sonnet on the scene, it seems we’ve got a new king of the hill.

According to Anthropic’s benchmarks, the new Claude 3.5 Sonnet takes the cake in just about every test against the other most capable frontier models. 

But how does it really stack up in real world testing? And should you start grabbing for the Sonnet shaker the next time you’re sitting at the LLM table? 

To get some answers, we pitted Claude 3.5 Sonnet and. GPT-4o against each other, head to head. 

Full disclosure, our real-time tests aren’t scientific or definitive. They’re just quick looks at how the top two models stack up. 

So, here’s what’s new in Claude 3.5 Sonnet, what we’re loving and hating, and how the two heavyweights stack up against each other. 

Ready? 

One, Two, Threeeeeeeeeeeeeeeeeee. 👇

In short, the upgrade from Claude 3 to Claude 3.5 Sonnet was pretty sizable.

Anthropic also introduced the Artifacts feature, a new side-by-side interface that helps you not only create but render code and visualizations in real time. (For all other LLMs, you’ve gotta go copy and paste code into a third party software to do this.) 

The Claude 3.5 Sonnet model itself is faster, cheaper and more capable in just about every way. 

Also worth noting — it’s only been 3 months since Anthropic released Claude 3, introducing three different flavors: 

Haiku — The fastest but least capable model 

Sonnet — The middle model 

Opus — The largest and most capable model

The takeaway: 

Also, let’s cut it straight on the naming and (probable) strategy here. 

By upgrading their middle Sonnet tier and not the top Opus tier, Anthropic is clearly firing off warning shots, in the same way that Meta’s smaller 70B models grabbed headlines while it’s still sitting on its larger offering. 

In short, everyone’s saving something up their sleeve for OpenAI’s next big offering.

Presumably, Anthropic will have Claude 3.5 Opus ready to trump in whenever OpenAI releases its next big thing. 

In the race for enterprise customers and outside funding, model makers gotta be nimble in their offerings and working models ahead.

That seems to be Anthropic’s approach here. 

There’s a lot of each, TBH.

Again, we’ve only spent a few hours in Claude 3.5 Sonnet, but here’s our first impressions of the good and the bad.

What we loved:

If you’re not a GPT-4o power user, you were probably blown away by the speed of 3.5 Sonnet.

Even compared to Opus, it’s Usain Bolt fast. The wild part is Opus didn’t feel slow until OpenAI released GPT-4o in May.

Ever since, Anthropic’s offerings have felt sluggish.

Not anymore. 3.5 Sonnet is faaaaast.

Also, the new Artifacts feature is a gem, especially if you code.

A built-in interface that renders code, designs and more in real-time is a much more intuitive way to work with models.

The biggest disconnect in working with models is what happens AFTER a generation. Maybe it’s a lot of copying/pasting/modifying, or taking outputs into other programs and testing it.

With Artifacts, a lot of that is gone. Being able to render a front-end design, or execute code, in real time feels fresh and much-needed.

What we hate:

Sorry, we still can’t honestly recommend the front-end Anthropic Claude experience for most business use-cases.

Why?

Claude remains the only major LLM without some sort of real-time access to the internet.

We’re sure it’s easier said than done, but it’s a legit necessity in today’s LLM game. Meta, Google, Microsoft and OpenAI all have different real-time capabilities in their respective models.

And even though 3.5 Sonnet has an April 2024 knowledge cutoff date, a not-connected model is a dangerous game to play in terms of accuracy, hallucinations, etc.

And a little more of what we hate in the head-to-head below.

Let’s reiterate the obvious — our head to head was not a scientific test.

We ran a bunch of simple, zero-shot prompts and used our best judgement to gauge which model had better responses.

With proper prompt engineering, we would have gotten better results from each model, but that’s not the point here. 

Also, we only compared similar features and used tests that would draw on those similar features. 

• So, we weren’t going to test the ability to render code, as Claude is the only one that can handle that.

• Similarly, we weren’t going to test recent events and trends, as GPT-4o is the only one that can handle that.

Also, we started each chat off by encouraging each model to take it step-by-step and to think critically about each prompt before responding. 

With that in mind, let’s break down where each model shined or fell flat. 

Claude 3.5 Sonnet

Speed – While the output speed is lightyears ahead of Claude 3, it was pretty on par with GPT-4o. So we’d call that a push.

Copywriting –  TBH, the only area where Claude 3.5 Sonnet shined in our very limited matchup was in its ability to create written content. Claude’s copy sparkled, whereas GPT-4o’s out-of-the-box style sounded bland and uninspiring. 

Long docs – And for all the hubalub about Claude’s extended/better memory, it’s harder to reap those benefits when its file upload capabilities are soooooooo limited. 

ChatGPT 4-o 

Logic — In most of our logic and reasoning tests, GPT-4o had a VERY slight advantage. While neither shined, GPT-4o in our tests was consistently either slightly more right or slightly less wrong. Lolz. 

Data Analysis — Where we saw GPT-4o shine, though, was in its ability to handle our data analysis tests. Claude couldn’t even handle a large spreadsheet, let alone crunch the data.

In this head-to-head, Jordan’s jaw literally dropped when GPT-4o crunched through more than a half million cells of data and started visualizing the data accurately in near real-time. Sheesh. 

Copywriting — Out of the box, ChatGPT is a terrible copywriter. We saw this in our head-to-head. (With proper prompt engineering though, ChatGPT can crush this. In a simple zero-shot head to head, though, Claude crushed ChatGPT.) 

So, which model do you think is better?