2024, week 37

Connecting Insights & Ideas

This week we are continuing to develop "Suggested Insights on Ideas".

Rough can now identify connections between Insights and Ideas.

This works both ways, Ideas now have a "Related Insights" section, while Insights have a "Related Ideas" section. Note that this section only appears if Rough has found a related insight/idea to show you.

Realtime Suggestions

We are also experimenting with updating your suggested connections as you type. This means as soon as you publish your new Insight or Idea, Rough will have a list of related insights for you to review.

Keeping Your Data Private

We respect the privacy of our customers - we want Rough to be a product you can trust.

There is a lot of hype around "AI", with software companies adding "magic" buttons and anthropomorphized chatbots. Sometimes these features can be useful, but they often feel tacked on as a gimmick, a solution to a problem that didn't exist. These features often come at a cost - your data being shared with a third party to do who-knows-what-with. Companies are releasing features and facing backlash from customers over these privacy concerns.

Our new Suggested Insights feature marks the first time where we've used AI technology in Rough. It's not magic - recommendation systems have been around for decades. But with the recent advances in machine learning, it's easier than ever before to develop a system that can find connections between a huge number of ideas.

We are using the mxbai-embed-large-v1 embedding model. Our initial implementation used the cloud-hosted version of the Mixedbread.ai API - allowing us to get up and running and verify that this approach would work.

This week, we are hosting our own private version of this API. This means that we can process your data without it ever leaving our server.

To make this possible, we have upgraded our server hardware. We were previously hosted on a small server in New Zealand. We chose to use the Mixedbread embedding model because it is openly licensed, has great performance and that it only uses 1.25GB of memory. However, even with this small model, our server was not up to running our existing architecture as well as this model.

Unfortunately, server hardware in NZ can be quite expensive. We (Jacob & George) are both based in NZ and when we tried hosting overseas we found the web app to be very sluggish (due to the network latency). A few months ago we rebuilt Rough with local-first support, giving a great user experience regardless of the network latency. This has allowed us to host the app overseas while still having a great performance around the world.

We are currently trialling a server in Europe -- with plenty of RAM, but not costing much more than what we were paying for our NZ server. This server has been able to easily host our infrastructure and the embedding model without breaking a sweat.

Technical Details

Our server currently runs four services:

PostgreSQL Database

Node.js Server

Tusd (for handling file uploads)

Imgproxy (for serving media files)

We are now using Infinity to

self-host a server to generate embeddings.

A rough overview of our architecture looks like this:

Next Steps

The work so far has been heavily focused on the underlying infrastructure. This week we hope to start improving the user experience by working with related insights/ideas.

This includes adding buttons to hide unrelated suggestions or pinning them to show that a human has approved that there is a real connection.