An idea is like a virus, resilient, highly contagious. The smallest seed of an idea can grow. It can grow to define or destroy you.¹

Bringing an idea to life is surprisingly difficult. While the world is now teeming with AI products vying for the chance to transform our raw ideas into executable code, there is still mountains of work to do.

I know this, because I spent the last 18 months building a product of my own. With the help of some human and AI companions, I built a security product I’m very proud of.² But this came at a huge cost.

For months, my life was spent wrestling with infrastructure, developing and refining APIs, writing data ingestion and transformation pipelines, integrating large language models for more advanced product features, making performance enhancements, fixing bugs, and more.

This summer, I was too busy to even take time for myself. No breather or break, just a sea of endless work.

Then a few weeks ago, one of my customers asked if I knew any Golang engineers. He said his team had spent three months building out a new product feature and the corresponding APIs, but progress was slow and bottlenecked.

These poor saps were just like me, chipping away at mountains of work, probably with the help of AI, but unable to escape the sheer vastness of the mountain’s shadow.

The problem

I think part of the problem is the level of granularity we apply to our work. Shipping a new product requires us to answer too many technical questions which are of little direct interest to our customers:³

• How should we ingest, transform, and enrich our data to maximize its value?
• Which storage engines should we use?
• How should we make our data available to customers?
  • REST, GraphQL, event streams, UI components?
• How can we quickly and safely evolve our public-facing endpoints and data models?
• How should we secure and segment data at a customer-level?
• How can we design systems that effectively support more sophisticated, AI-powered features? e.g. supplementing LLMs with knowledge from external sources⁴

The answer to many of these questions will likely change over the product’s life cycle, and that’s unfortunate because all we wanted to do was bring an idea to life, but now we’re worried about so many other things that threaten its eventual fruition. But are things getting simpler?

In-roads

Luckily, there are many tools out there that eliminate some of the toil of bringing an idea to life and sharing it with the world. Vercel’s v0 is a great example of streamlining UI development, and Supabase is a notable example of simplifying the persistence layer with automatic API generation for Postgres tables and making user authentication not terrible.

This doesn’t even begin to cover advancements in more specialized layers, including Redpanda’s innovations in the streaming space, Clickhouse making data warehouses blazingly fast, and DuckDB packaging a full-blown relational database that can be embedded almost anywhere.⁵

But for many people, integrating these disparate tools into a cohesive, user-friendly product often requires significant effort and expertise, creating a barrier for many aspiring innovators and creating slow development velocity at many companies.

Data Products

Imagine for a moment that we could set aside the lower-level implementation details. What if the most technical and time-consuming questions had already been addressed for us, and the functionality derived from these solutions was neatly packaged into a single, higher-level entity known as a data product (closely related to the data mesh concept).⁶

Consider the following example. Say we’re creating a new AI automation framework and want to store customer-level information about their various job runs, performance statistics, and other observability measures for their AI pipelines.

Instead of going through the ceremony of setting up the required database tables and Kafka topics, wiring up the service endpoints and auth layer, and writing the queries that ultimately retrieve the data for our front-end, we instead create a new instance of a data product that includes all of these things, supported by pre-vetted technologies at every layer.

beachday create data-product --name job-analytics --team platform

········································ 100%
Created 'job-analytics'.
https://job-analytics.beachday.ai/

When you create a data product, deployment happens immediately. It’s not some final step or milestone that happens after you’ve ironed out all the minutiae of your idea. It’s existence isn’t delayed all; it immediately comes up life the second you want it to, as canvas to build upon.

And this simple, powerful entity has everything we need to start building our product features:

• Ingestion endpoints for getting data into the system
  • Simply POST data to any number of table endpoints to get data into the system:
    CURL -XPOST https//my-data-product/{table} -d '{"key": "value", ...}
  • Or attach external data sources if your data already lives in another system (Kafka, Redpanda, Postgres, S3)
• A serving layer for getting data out of the system
  • Create composable and version-controlled SQL pipelines to extract, aggregate, and transform the data
  • Publish the results of these queries as endpoints
• A security layer for protecting the data
  • Create, rotate, revoke, and suspend API tokens for your customers
  • Automatically filter data at the customer level using row-level security
• Collaboration features for iterating quickly and safely
  • Create branches of your data product, and submit changes for your team to review
  • Easily share your data product with other teams for internal use

This is the vision of our platform. It allows us to shift our attention from the ‘how’ of building products to the ‘what’ and ‘why’. These questions are more relevant to our customers and the ultimate litmus test of our ideas. Our platform replaces the laundry list of technical questions presented earlier with a new set of customer-centric questions:

• What data does the customer need?
  • Not how do we get it into and out of the system
• Who will need access to our data?
  • Not how should we manage access and security
• Which teams will own the data product?
  • Not how do we manage data governance
• Why is this product valuable to our customers?
  • Not how is it built?
• What insights can be derived from this data?

For me, the problem is worth solving, because the current approach drains our most valuable resource: time. So I’m going to give you what I couldn’t give myself the first time I built my own software platform: the ability to focus solely on your idea. A chance to protect your most valuable time. A vacation from the mundane and tedious. I’m giving you a beach day.

The goal of Beachday is to streamline the process of building data products. We can achieve this by:

• Bringing a best-in-class data serving layer to you
• Making data ingestion, enrichment, and transformation simple and composable
• Giving you an adaptable and isolated service layer for each data product that runs at the edge, takes seconds to deploy, and can be used in RAG use cases or traditional software applications
• Auto-generating API endpoints and UI components so you can build beautiful user-facing products with ease, giving your customers a plethora of options to access your data
• Most importantly, replacing lower-level technical concerns with higher-level primitives to help you bring your idea to life

Achieving these goals means more engineers and companies will be able to materialize their visions, whether for a small startup or a large enterprise. It will mean saving our most precious and limited resource: time. By extension, that also saves us money.

If you have opinions on these ideas, or if you’d be willing to be an early adopter or beta user of our platform, please let us know. As for the product I’ve been building at Honeypot, this will be integrated into the larger Beachday platform as a turnkey, high quality data source for analytics use cases, and as we migrate it to Beachday, it will be the ultimate litmus test of our own ideas.

Thanks for reading and being a part of this journey.

• Mitch Seymour, Founder

Endnotes

Footnotes

1. Inception, 2010 ↩

2. The product is called Honeypot, and will be integrated into Beachday as an analytics starter kit. More on this in a future post. ↩

3. Execution is extremely important, but these are implementation details from a customer perspective. ↩

4. https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/ ↩

5. Can we please get DuckDB support in Cloudflare workers? ↩

6. https://martinfowler.com/articles/data-mesh-principles.html ↩