Make a devtool agents want

I created Preseason.ai because I was curious about how much power current LLMs have in recommending developer tools to people vibe coding their own web apps.

Vibe coding is obviously in a crazy growth phase, with some companies going from zero to hundreds of millions in revenue in less than a year. And with that, they are also pushing along a set of specific dev tool companies with a similar trajectory.

So I wanted to track this and built an open-source benchmark for it.

Every day, Preseason sends prompts to the top open and closed-source LLMs from the main labs and measures which developer tools they recommend.

For example:

  • What database do they recommend?
  • What authentication provider?
  • What UI framework?
  • What email provider?
  • What search provider?
  • LLM Observability
  • And so on.

It has been very interesting already.

One example is email.

As you can see below, there is still a lot of competition in this category, with SendGrid and other legacy providers still showing up strongly.

Preseason rankings for email API recommendations

But you can also clearly see Resend as a modern startup that is very well designed, has great documentation, and seems to be growing very fast.

I would not be surprised if a meaningful part of that growth is now coming from LLM recommendations.

Imagine someone starting a new SaaS for a small business right now. They need to send emails to their customers. A lot of the time, the LLM is probably just recommending Resend and sending them a new customer.

That is pretty cool.

Another example is databases.

Preseason rankings for database recommendations

Another example is search.

Preseason rankings for search recommendations

Another example is LLM observability.

Preseason rankings for LLM observability recommendations

The project is open source, so please feel free to open a PR if you have a good idea for how to make the measurement more robust. I will keep tracking this as I go and post updates from time to time.