TALKS
S
S
U
U
B
B
S
S
C
C
R
R
I
I
B
B
E
E
T
T
O
O
N
N
E
E
W
W
S
S
L
L
E
E
T
T
T
T
E
E
R
R
TALKS
S
S
U
U
B
B
S
S
C
C
R
R
I
I
B
B
E
E
T
T
O
O
N
N
E
E
W
W
S
S
L
L
E
E
T
T
T
T
E
E
R
R
A
A
L
L
L
L
T
T
A
A
L
L
K
K
S
S
Keynote
Modern Inference for Modern Workloads
Abstract Coming Soon
Speakers
Bryan Bischof
, Head of AI, Theory Ventures
Charles Zedlewski
, CPO, Together AI
Tuomas Rintamaki
, Research Scientist, NVIDIA
Charles Frye
, Developer Advocate, Modal Labs
View transcript
00:00
Welcome to this discussion.
00:03
Okay, so
00:05
my name is Brian.
00:08
I don't really understand inference systems, which is why I hosted the inference systems track yesterday and why I want to do this panel.
00:16
But I think the thing that I most not understand is what application layer developers need to know right now.
00:22
And so that's why these people are here to help me help you understand that better.
00:29
I want to start out with a really sort of like warm introductory question to keep us on sort of like level ground.
00:37
Is fine-tuning dead?
00:39
That's a great one. I can start. I think
00:45
fine-tuning is not dead. I think we just call it RL now for some reason. We should say like
00:53
we see, you know, I work at the infrastructure layer, work with a lot of different companies that
01:01
are deploying open source models. And in many cases they are
01:06
running their own reinforcement learning, often, you know, in
01:10
interaction with a compute environment in an agent sandbox or something like that.
01:15
And so it's it's not
01:18
fine-tuning that people might be familiar with from a couple of years ago. It's not LoRa fine-tuning.
01:24
It looks a little bit different and meaningfully different. Like the system is like very, very different.
01:29
The compute needs are very different. The frameworks that you need are very different.
01:33
And there's a difference between like collecting data versus defining an environment.
01:38
But those differences and the terminological difference, I think, obscure the fundamental
01:45
like
01:47
similarity, which is that you take a highly capable foundation model and you build something on top of it.
01:52
But OpenAI just, you know, they just announced that the fine-tuning API is going away.
01:57
Yes. Isn't that what we need to know? They also announced that Sora is going away, but video generation is, I assure you, not dead.
02:06
Interesting, interesting.
02:07
I'm curious how NVIDIA thinks about fine-tuning, especially with the Neumatron family.
02:13
Yeah, right. So, yeah, I work on the Neumatron Open Foundation models from NVIDIA.
02:21
And we release, obviously, the weights, but we release also data sets and tooling for people to fine-tune these models.
02:33
So, yeah, we see that fine-tuning is not dead.
02:37
We are actually working with a bunch of customers who are taking our models and they have specialized use cases in specialized domains.
02:48
One example is document intelligence, for example, understanding documents that they have received, things like that.
02:57
And they really like to take open models and fine-tune them for their use cases and domains.
03:04
Why do you need to fine-tune for document understanding?
03:06
These multimodal models are so good.
03:08
They're so sort of like VLM in a box and everything else is there.
03:12
Why would I need to fine-tune?
03:14
Right. So what we are seeing that some of these customers, they have like specialized domains and they also have internal evals.
03:24
And some of these frontier models, for example, they are very good generalists, but they are not necessarily doing very well in those specific domains.
03:34
And these customers, they may also have like their custom data sets.
03:40
And they find that fine-tuning open models can lead to even better results than the frontier models at a much lower cost.
03:51
What kind of together customers are fine-tuning?
03:54
A high percentage of them.
03:57
And it's funny, a year ago, I actually felt like I heard more of the fine-tuning is dead than I have heard recently.
04:05
So like a year ago, the ethos was, yeah, I thought about fine-tuning this model, but then I just waited for GPT to get better.
04:12
And if I just wait, that seems to take care of it.
04:16
And so there was sort of like a decline in interest probably about nine months ago, and then it's been building as of late.
04:25
The, let's see, what's the best way to synopsize this?
04:28
So one of the major advantages of fine-tuning, whether it's supervised fine-tuning or RL fine-tuning, we'll kind of put them all in the same general.
04:38
Together, we have all those under one team we call model shaping.
04:40
So let's just say like, why do you want to do model shaping?
04:44
One is you might have some like super narrow domain that there's not a lot that's not in part of the regular distribution of any of the, whether it's open or closed-weight models that you want to have access to.
04:57
And so you can simply get a higher quality of experience by adapting to your specific domain.
05:01
And that might need specialized evals or unique data that wasn't part of the training set for the public models.
05:07
I think the bigger payoff though is it's the ability to compact the amount of intelligence you need for a feature of your application into a much smaller form factor.
05:17
And that actually, that turns into dollar savings, but I actually find that most people aren't doing it for dollar savings, they're doing it for user experience.
05:25
Because when you compact the type of intelligence you need, so for example, one of our customers is Decagon, there's all kinds of steps that go into the customer support process they run.
05:34
Anytime they can take one of those steps and compact the required intelligence and not give up on quality, but put it in a much smaller form factor, they reclaim latency.
05:43
And to reclaim latency means that that one interaction that somebody has with that agent, you can now add more features, right?
05:49
So if you like a fixed latency budget of a few hundred milliseconds to get something done, you might be able to inject two or three more steps that the AI is doing for that call and fit within the same latency budget.
06:03
And it's basically because you compress the requisite intelligence into many smaller models.
06:08
At this point, Decagon, and this is only one example, will proudly say that I think not a single model left that they use in their application is vanilla.
06:17
They have all been fine-tuned at one degree or another.
06:21
So kind of a follow-up on that, like, as we move towards this, are we seeing a difference in the inference mix or sort of like the mix of things that people are trying to do with the models?
06:32
Are they trying to do quite different things than they used to be? Or is it more that we're just getting better as an industry and getting more aware of the sort of what's available to us?
06:42
I mean, I think the big change in mix happens, whatever it was, like, it depends where you want to mark it, maybe December last year or October last year, where we sort of went from this chat orientation to more of this kind of long-horizon tool-using agent orientation.
07:00
And I think that also is what changed some of the demand for fine-tuning.
07:04
Before, when your applications were very chat-centric, I think there was a much more limited set of domain adaptations, and you couldn't really compact an entire chat experience because you kind of need a strong generalized base to have a good experience.
07:17
But now, when you start talking about various forms of automation, whether that be back-office finance or whatever, content production or video production or whatever, all these kind of like multi-step tool things, now that's why RL is on the rise, because you're like, well, I have my specialized path for my specific process, and I want to RL my model until it just fits that need, working off like a strong but smaller base.
07:44
So I think that kind of the long-horizon agent inference pattern, that's the thing that's like dominating the growth for this year.
07:53
And I think also part of what revived, certainly generated more interest in RL and probably also grew.
07:59
demand for bottle shaping generally.
08:05
Sort of in the same category, I wonder about sort of the heterogeneity of these requests
08:09
because we get these, let me give you an example.
08:12
When I'm coding with agents, I get quite heterogeneous requests and I get quite homogeneous responses.
08:21
And so I will ask things that feel small and sort of like I expect a small answer and a
08:28
small change.
08:29
And I ask things that are big and I expect a big answer.
08:33
And right now, my experience, especially with the frontier models from the three core labs,
08:38
maybe four core labs, those models feel like they are surprisingly homogenous in their
08:43
response.
08:44
And so I'm curious, who's responsible for engineering this in?
08:49
Is this the agent harness that is needing to slurp up this awareness?
08:54
And what does this mean for the inference providers?
08:57
Do you all sort of expect different access patterns of the inference because of this?
09:03
Yeah, I can start.
09:07
I would say we see a lot of people deploying multi-model inference systems where there
09:12
are sort of models of different capabilities or different specializations that are perhaps
09:18
orchestrated as tool calls or sub-agents or whatever from some top layer intelligence
09:27
and that helps you shape the amount of compute and the expense and also the latency in order
09:37
to respond to the different kinds of user queries.
09:42
And I think the pattern that we've observed is that people sort of start as they move
09:47
from using purely proprietary APIs, they start with those sub-agents or tool calls as the
09:53
first thing that they fine-tune because they maybe already are using a generation back
09:59
of model or they're using the smallest model in the lab series and so they have more confidence
10:05
that they can get a small model to perform well.
10:12
Once you move away from the idea that there is one single language model that handles
10:16
all user queries, you start sort of like breaking it up and then being able to sort of have
10:22
a more heterogeneous response.
10:24
Yeah, maybe like an even sharper version of this question is, if you're building an agent
10:29
for your customers at the application layer, are you responsible for model routing or do
10:34
we expect as we move forward that the inference provider is going to be responsible for model
10:39
routing?
10:40
No, I think that's it, so yeah, I feel very strongly about this one.
10:44
The closer you are to a domain-specific tool or application, the more this is your responsibility
10:52
and your great opportunity because essentially, if you're an agent builder or an application
10:59
builder, it's kind of like what you're really ultimately doing is contextualizing intelligence
11:03
in a domain or a class of user or something like this.
11:06
Your reward for that is that you may have started with the biggest, fattest, most expensive
11:10
model at whatever that token cost is, but as you get the benefit of watching the usage
11:15
of your application, you're able to characterize what intelligence is actually needed, decompose
11:20
it into its requisite pieces, and by the time you break it down into the constituent
11:24
models that actually need to serve it, the spread between what you used to pay and what
11:29
you pay now is enormous, and that actually represents defensibility because if some other
11:35
yokel decides to get into your market and build a similar application, say, how hard
11:39
can it be to prompt Opus, not very, but to accumulate a bunch of users and to understand
11:48
what mix of intelligence actually did the best and break that down into the submodels
11:51
and adapt it, that took a lot of time and effort, and it's also very difficult for anybody
11:56
else to see and copy, so I think it's one of the number one rewards for being an application
12:03
developer.
12:04
The trouble with us as an inference provider, and we've actually run experiments with this
12:08
where we've tried to do auto-routing and these kinds of things.
12:10
We have a lot of research engineers on the team, and the problem is we are the most general
12:15
type of, you know, we take all commerce, right, in terms of the nature of the inference we
12:20
supply, and so we have the least ability to characterize a pattern out of anybody.
12:26
Everyone that's one click closer to a particular domain or user application is going to be
12:31
more effective at this than we are.
12:34
I think the very base technical level of like sort of routing without any thought to which
12:38
one you want, but just sort of like policy, cost collecting, and that type of thing, that'll
12:43
be a generic layer, but everything else is not a chore.
12:47
I think it's the reward you get from being a successful agent or application developer.
12:51
Yeah, and actually just to follow up on that, and also draw a connection to Charles' response
12:58
on fine-tuning, like the concern, you know, a year or two years ago was like, oh, why
13:04
would I fine-tune when I can just wait for a new model to come out and it'll be better?
13:08
Brian and I were big eval boosters at the time and said that like the goal, your goal
13:15
should be to collect up the system that allows you to evaluate and improve models over time.
13:20
That is like, that's the true machine that you are building.
13:24
It's not the like lines of code that you are writing.
13:26
It's like the engineers, the data, and the code that together build that system, and
13:32
I think we've seen that like really come to fruition over the last couple years as
13:37
people like construct model routers, they fine-tune smaller models, they develop reinforcement
13:44
learning environments, curate fine-tuning data sets, and otherwise like build this like
13:49
thicker substance of what it means to like build these intelligent products.
13:55
Ultimately, you're talking about the sort of moat that you can build at the application
13:58
layer of seeing what people want to do.
14:00
Yeah.
14:00
I'm curious as you've been training a lot of these models and making them very good
14:04
at different things, and in some cases specific things, what's the kind of like, you know,
14:10
golden egg?
14:11
Like what's the data that you're looking for that you're like, this is going to improve
14:14
the model?
14:18
So, I think we like to work with our customers and learn from their experiences and from
14:27
their evaluations, and learn especially about cases where the current models are failing
14:35
for them and trying to solve those kind of cases.
14:39
So, for example, we recently released this model called Nemotron Omni, and I mentioned
14:46
the document intelligence use case earlier.
14:49
So, some of our customers had like customized pipelines where they were using like a combination
14:55
of models like OCR model for parsing documents, and then an LLM for understanding the documents.
15:04
So, models like Omni, they can basically simplify the deployment, have just one model, and
15:14
understand documents directly.
15:16
And whenever you have like combinations of multiple models like that, then the handoff
15:22
points between the models, they can introduce some kind of like loss in accuracy and loss
15:27
in context, whereas the Omni model, for example, doesn't have that kind of issue.
15:34
Omni is a pretty general and like multimodal model.
15:36
Should all the models be multimodal?
15:38
Should all the models be general?
15:40
Should we expect extreme speciation in the models that we're using?
15:45
What's your take?
15:45
So, I think probably not, because if you are dealing, for example, with a text-only
15:54
use case, then it's much simpler to just deploy an LLM.
15:59
And it's going to be cheaper to run an LLM versus deploying an omni-model
16:05
that uses more GPU memory and is harder to deploy.
16:09
So I think it will be still use case specific.
16:13
Got it. Got it.
16:15
Kind of in that same theme of like speciation,
16:18
we started with only a few models and we've now gone to
16:22
there are quite a lot out there in terms of like what you can choose.
16:26
Do you all expect a sort of big crunch?
16:28
Do you think that we will go back to sort of like leading candidates and
16:32
strong horses and then fine tuning off of a smaller set of swim lanes?
16:37
Or do you think we will continue to, you know,
16:41
see more variety in the population of models?
16:46
I think you want to just,
16:48
I think maybe it makes sense then at this point,
16:50
because we talked so much about model shaping and adaptation,
16:53
that you want to distinguish between like the,
16:56
what's the adaptable base and then what's like the finished model.
16:59
So for finished models, no, I think they'll continue to proliferate.
17:02
They'll be more on hugging face, you know,
17:04
some order of magnitude more in hugging face in the future than there are today.
17:08
But if you look at all those models,
17:09
there's a finite number of base models that they tend to be derived from.
17:12
There aren't like an infinite number of model architectures and sizes and what
17:15
have you.
17:16
So if I think about like how many base models ought to be there in the world
17:20
today,
17:21
probably the number will trim down a bit from where it is right now.
17:25
I don't know what the equilibrium state is.
17:27
It's not going to be, you know, like the, you know,
17:30
three proprietary model labs or something like that.
17:33
It's going to be something more than that.
17:35
But like, if you look at like, you know,
17:38
sparse MOE like models that are good for,
17:42
for multi-step reasoning, like there's,
17:45
that are like between 500 billion and a trillion parameters.
17:49
I think you've got like a dozen, not 10 or a dozen options right now,
17:53
but probably like five or six would do.
17:56
So there'll be some thinning out, but it doesn't,
17:59
I don't think that's going to come anytime soon.
18:01
These things are all too well-funded right now.
18:03
Yeah.
18:04
I think that's part of where that question comes from is just like,
18:06
what's the guy to do?
18:07
Like an application layer developer like me, like, I don't know,
18:10
like I don't know which of these models I should be testing and trying.
18:13
It feels laborious.
18:14
And if you add the like specialization layer,
18:17
like if I'm going to model shape each of them, like, okay,
18:20
this is like adding quite a lot of complexity. Yeah.
18:23
Yes.
18:24
But if you engineer your model shaping layer appropriately,
18:29
then swapping out the models shouldn't be too hard.
18:32
Like obvious, I mean,
18:34
if you think like inference code is hard to work with,
18:38
wait until you try out any of our beloved training frameworks that are out
18:42
there. But so, so it's a little janky. It's a little, it's, it's,
18:46
it's PhD student code in a lot of cases.
18:49
And I say that as a former PhD student. But yeah, you, you,
18:54
the key thing is that you, the, the data, the environment, the,
18:57
like the stuff that shapes the model shouldn't change too much. And,
19:01
and it's, it's more like, you know,
19:04
sort of other ways that people slightly rearchitect or,
19:07
or change features in their backends today and less like migrating from
19:12
Postgres to Oracle or, or some,
19:14
or these other sort of like large scale people,
19:17
like there's deep lock in and it's really hard to get out. So I,
19:20
I wouldn't worry about that too much as like as an application layer
19:25
engineer looking at, at running your own inference.
19:27
I think it's probably better to,
19:29
to maybe pick a model that does well on sniff test and,
19:34
and is beloved on local llama or Twitter or wherever else you get your,
19:39
your taste from and stick with that for a while and,
19:44
and not worry about chasing like five extra points on humanities last exam or
19:49
whatever,
19:50
or 30 extra minutes on meter or whatever people are excited about.
19:54
Cool. Yeah.
19:55
I think these things are,
19:56
these are more fungible than any other kind of infrastructure I've seen so
19:59
far. Like it doesn't, it's perfectly fungible, but, but like I, I,
20:03
there's a young startup and I was like, Hey,
20:05
we have this like new approach strategy for test time.
20:07
Computers should like give you like a big boost for your specific use case.
20:10
This startup has like 10 employees and like, Oh, no problem.
20:12
We've already got this layer set up and like,
20:14
it's going to let us like try different models.
20:15
We'll plug you in and it'll take a week.
20:16
We'll decide. And if it doesn't work, we'll part ways.
20:18
Like it's, it's surprisingly efficient. That's great.
20:21
That's great. Yeah. And I guess for selecting models,
20:24
I would still use the public benchmarks as a heuristic.
20:28
There's a lot of like very specialized benchmarks for very,
20:33
very specific tasks.
20:35
And if you go look at the leaderboards for those benchmarks,
20:38
that can at least give you like some signal of what models are going to be
20:42
good in that domain.
20:44
We usually see a correlation between the public benchmarks and like private
20:50
evals. So that can be useful to look at.
20:54
Yeah, absolutely.
20:55
I definitely agree that they're well correlated, just not perfectly.
20:58
I want to pivot a little bit because something came up earlier that I think
21:03
I want to spend some time on.
21:05
And that is sort of this like sea change around like what people were doing
21:10
with tokens, like where are the tokens going?
21:12
We very lightly touched on the fact that like with longer running agentic
21:17
tasks, specifically like coding agents,
21:19
there has been a change and more and more people are firing more and more
21:25
sort of, they call it the slot machine.
21:28
They're pulling the slot machine more and more.
21:31
And I think one of the things that occurs to me is that we will see a lot
21:37
different expectations about how inference behaves.
21:42
If you have lots of agents running in sandboxes,
21:46
sort of like kicking off these ephemeral environments where they're going to do
21:50
stuff,
21:51
that's going to look very different than you've got one agent that you're
21:54
talking to and it's hanging onto this long conversation.
21:57
And so I think we're probably all in alignment about like tokens go up,
22:03
but I am curious in terms of like, what's the shape of that function?
22:09
Is that the good old exponential or are we looking at something even more than
22:15
exponential here?
22:16
For the growth?
22:17
Yeah.
22:18
And the way that it grows, because I know myself,
22:22
I remember, you know,
22:23
doing a couple billion tokens in November of last year and now I can do a couple
22:28
billion tokens accidentally.
22:33
Yeah.
22:35
I have a hard time forecasting like just what the, like how, okay.
22:39
All I know is this.
22:40
I know that like right now the, the, the,
22:44
the token usage has gone up something like 10 X year over year.
22:50
And when I've done any kind of informal surveys,
22:52
it feels like the vast majority of that is attributable to coding agents.
22:56
And when I look at like all the software developers in the world that are
22:59
actually like fully embracing coding agents,
23:02
all the estimates might pull people is like somewhere between five and 15%.
23:06
So, and then you realize, well, that,
23:08
that same long horizon pattern is good for just a ridiculous amount of other
23:13
knowledge worker kind of wrote,
23:15
wrote tasks that I think collectively are probably many multiples larger than
23:21
coding agents.
23:22
So I just know that we're just extraordinarily under penetrated.
23:26
And that's why, like, if, if you're an inference provider today everybody,
23:30
I think,
23:31
starting from the beginning of the new year has just felt this like giant
23:34
sucking sound, right?
23:35
Like in everybody is tapped right now and everybody is maxed out on capacity.
23:38
So everybody are undershot where, where we're going to be at.
23:42
And when I look at the supply side of it I don't see how supply grows at the
23:50
same pace as demand for at least the next three years.
23:54
So that doesn't give you, so yes, the demand is exponential.
23:57
Like how exponential?
23:59
I couldn't quite tell you, but I don't see any way where all the places where you can
24:05
find more efficiency in the way you produce supply, I just don't see how they add up to
24:11
more than the demand.
24:13
Yeah.
24:14
Yeah.
24:15
I fully agree with Charles on the agentic token usage, but a big focus area for us has
24:22
actually been efficiency.
24:25
So the omni-model that I talked about earlier, it's very good in video understanding.
24:34
And videos, they consume like a huge amount of tokens.
24:40
And we wanted to make our model as fast as possible for video understanding.
24:46
So we actually developed a ton of techniques for token compression and token reduction.
24:55
So that could be another sort of factor at play that people start developing like specialized
25:02
techniques for reducing and compressing tokens.
25:07
I'm glad you brought this up because, you know, there was a period of time where we
25:11
didn't have like prefix caching, and then we got prefix caching, and there were a lot
25:15
of people asking, okay, how do I optimize my system to take advantage of prefix caching?
25:20
And now I hear people asking a similar question, but almost like at a bigger scale.
25:24
How do I optimize my system for sort of like token efficiency?
25:27
What are the other tricks that I can build?
25:29
Is this a, again, coming back to the question I asked earlier, is this a harness engineering
25:33
question where I'm the application builder, I'm responsible for figuring this out?
25:37
Or should I be expecting the inference providers to say, no, no, no.
25:40
We will help you reduce your token waste.
25:43
I think we've already seen with people using, say, different harnesses with the cloud models
25:53
or cloud code with different backends that the default behavior of the harness does things
26:01
like accidentally bust the cache on every single request.
26:05
And it's as simple as changing a header or an identifier that shows up at the beginning
26:12
of the system prompt, right?
26:14
And if you aren't sort of inference conscious, right, if you don't know a little bit about
26:19
how inference works, you can very easily make these mistakes.
26:22
So I would liken it to sort of the way that people interact with databases, where like
26:26
quite many people use managed databases and very few engineers, even at companies that
26:33
run their own databases, are managing those databases.
26:36
However, lots of people know that when you, like, if you write a SQL query that selects
26:42
a small number of rows, you probably want there to be an index that supports that query
26:46
if it's happening a lot.
26:47
And that kind of, you know, like, that kind of like basic knowledge of inference efficiency,
26:53
I think is important for, you know, building these, your harnesses, your applications and
27:01
making sure that they're like performant and efficient.
27:05
And obviously that extends as you control more of your inference, you have more opportunities
27:09
to do that.
27:10
Yeah.
27:11
So what are the top three coda?
27:12
Like, what do we need to not do?
27:13
What do we need to stop doing wrong?
27:15
Yeah.
27:16
Yeah.
27:17
Don't don't bust the cache.
27:20
Use the tensor cores.
27:22
That's number two.
27:23
So that's one where it's like, I think once you control your own inference, you realize
27:27
that there are opportunities for things like test time, compute scaling, like, like same
27:31
prompt multiple outputs.
27:33
If you have a good verifier, a good evaluation system, you can, like, bring that out of your
27:39
model evaluations and into your hot loop.
27:42
And that takes much better advantage of the hardware.
27:46
Yeah.
27:47
Those are my top two, I think.
27:49
Yeah.
27:50
And yeah.
27:51
To the previous question, don't go multimodal unless you have to.
27:55
Videos are a lot of tokens.
27:56
Yeah.
27:57
I think there's so many levers to pull to be efficient.
28:03
And it's going to be a shared responsibility between the application developers and the
28:09
inference providers.
28:11
And so, like, for example, my earlier example, the closer you are to the domain, how you
28:15
decompose your application into what types of intelligence, very hard for us as an inference
28:19
company to provide a lot of value there.
28:20
Like, it's just it's just too close to what you know.
28:24
But conversely, I actually think we do this today for some of our customers.
28:30
There's an opportunity for the inference provider to take a level more responsibility than the
28:34
database company did, because the database company would kind of say, well, I'll give
28:37
you this database and it'll be up and pre-configured and, you know, like, resize itself when things
28:42
get busy, which is all great.
28:46
But I'd say in inference, it's what you can do instead of say, for this class model, I'm
28:52
going to give you 50,000 tokens per minute with this with this individual per user token
29:01
per second experience with this tail latency.
29:04
And this will be your daily charge for that or your monthly charge for that.
29:08
And so that means like a lot of that kind of performance tuning stuff and that kind
29:12
of if you go out like with like a Snowflake or a highly or an Aurora or like a fully managed
29:17
database, you're still responsible for like your schema design, how you wrote your queries,
29:22
like like certain configurations that are unique to like the shape of your data.
29:27
And I think that actually inference providers, we can actually encapsulate a lot more than
29:31
that, which is a fantastic opportunity for us, because it means like if we believe in
29:36
the quality of our innovation in our own stack, it'd be real simple.
29:39
Like here's our here's our price for 50,000 tokens an hour.
29:44
And with these performance qualifiers, who can do better?
29:46
Yeah.
29:47
15 percent of the audience is excited that we're talking about query planning.
29:50
They're like, yes, my moment.
29:54
The number of recovering database people in the inference world is quite a few.
29:59
I almost I almost want to push this a little bit further because I want to know the first
30:04
principles economic argument for if the inference providers that are not the model providers
30:11
should be more token efficient price wise than the model providers.
30:16
Like what is the first principles economic argument?
30:19
Why I would expect this layer to actually be more effective price wise.
30:25
There's sort of this like larger like generation stack and there's a bunch of and and and so
30:32
like we're we're intelligence providers, but so are the proprietary model labs.
30:37
But we're working on where our innovation is at different layers of the stack.
30:41
And so we wind up with kind of like a different result.
30:43
So if you're a frontier model lab, the big distinction you have is the quality of your
30:49
tokens.
30:50
You're like, look, like I'm the absolute best benchmark on the outer edge of quality and
30:54
they charge a four to eight X premium for that.
30:58
And so if I'm if I'm a model for frontier proprietary model lab, like what's the number
31:02
one thing I'm focused on preserving that price premium, right?
31:06
Like that's that's worth more than, you know, we find all kinds of great innovations and
31:09
efficiencies that together we haven't found an eight X.
31:13
So like I think that's what that preoccupies them.
31:17
But now so now we know that these open weight models are about a quarter to an eighth of
31:21
that price.
31:22
And so if you're somebody like us, you're starting from that as your raw material.
31:26
So like we can work with the same open weight models as NVIDIA or modal and like many others.
31:31
So how are we going to distinguish ourselves?
31:33
So all of our R&D energy goes to these other layers of the stack.
31:38
And so for us, it's like whether it's engines or kernels or spec decoding strategies or
31:43
elaborate pre-filled decode decompositions, like we just wound up building that muscle,
31:50
whereas the proprietary model labs built a different muscle.
31:54
Another on this economic train of thought, I'm also wondering.
31:59
like, is Apple the next great inference provider?
32:04
We're getting better and better small models,
32:06
fast models, open source models
32:08
that I can run on Apple Silicon.
32:11
There's a reason why Mac minis are sold out
32:13
and it mostly has to do with lobsters,
32:15
but not only lobsters, you know?
32:17
Like, to what degree should we expect that 15, 20,
32:23
30% of our inference is gonna happen,
32:25
like, you know, on our desktop or on our,
32:28
like, on the desk in front of us?
32:31
I'd be curious, given that you build models,
32:32
what you think.
32:34
Yeah, well, personally, for example,
32:37
I've been running the Neutron Omni model on my Mac laptop.
32:41
So it's a great device for running that model.
32:47
And yeah, likewise, if you have like a GPU in your box,
32:51
then you can deploy open models on your local machine
32:57
and they run great, yep.
33:01
Yeah, but when you run it on your Mac,
33:03
what do you run it with?
33:05
What do you run the model with?
33:06
I'm using an app called LM Studio.
33:09
Okay, okay, nice, nice.
33:11
I do think, yeah, this is like the most cloud-centric
33:18
that inference is ever gonna be.
33:22
The lineage of this stuff is in, you know,
33:27
high-performance computing, right?
33:29
It's in these like hero runs.
33:31
And so they've been deployed in the same way.
33:35
But the, like, the demand for, like, local intelligence,
33:40
like, either low-latency intelligence or intelligence
33:43
where you don't have to actually think about, like,
33:45
the metering of it.
33:47
And it's just like, yeah, I have a computer
33:48
and it does this thing.
33:50
There's just tremendous demand for this.
33:52
I mean, yeah, I work at a cloud provider,
33:54
so it's not that I'm excited about this.
33:56
It's just the way that things are.
33:58
I think, like, it will remain the case
34:00
that there are reasons to have data centers, mainframes,
34:04
or whatever we call them in the future,
34:06
where people do this, the more intense work,
34:09
the work where you can, like, batch together many requests
34:13
and amortize effort more efficiently.
34:15
But yeah, I think in, like, 40 years,
34:21
like, that question will sound silly
34:22
because there'll be inference happening
34:24
in, like, little hunks of silicon all around every room.
34:28
And so the idea that they would all happen
34:29
in a data center will feel as, like, quaint
34:32
as, like, punch cards.
34:33
Is the data center in the room with us right now?
34:36
I'm not at liberty to answer that question.
34:39
I have one last question for all of you.
34:42
And this one's kind of off the record,
34:43
but do you guys know where I can get some GPUs?
34:46
Oh, yeah, yeah, actually, just get in a car right now
34:51
and drive, throw your cell phone out the window.
34:54
Actually, give it to me
34:55
so I can look through your list of GPU contacts.
34:58
Yeah, and then, yeah,
35:00
at a fair about eight hours or so, I'll let you know.
35:02
Okay, sweet, thanks.
35:04
Let's thank our speakers.
35:05
Thank you.