The implementation—evaluation loop

When an artist paints, they engage in a dance of two critical actions: meticulously applying paint to canvas up close, then stepping back to assess, contemplate, and strategize. This rhythm of execution and reflection is not unique to painting but is intrinsic to various forms of craftsmanship, including programming and writing.

Effective tools empower users to fluidly navigate between these phases of doing and reviewing at their discretion.

However, my experience with chatbots illustrates a challenge in maintaining this balance. Interacting with a chatbot entails a cyclical process of posing questions (execution) and parsing responses (reflection), which disrupts the potential for a seamless “flow” state. Each pause, waiting for a reply, is akin to the disruptive wait times encountered in programming during extended compile cycles, which can break concentration and disrupt the creative or problem-solving process. This interruption is equally evident in the use of chatbots, where the anticipation of responses can fragment focus and impede the fluidity of thought.

I think we have to create ‘tools’ that shorten the implementation—evaluation loop.

Autocomplete, realtime advice based on context, AI as a new sense for the world.

Any ideas?

2 Likes

I totally agree! I think we will need a lot more compute available to distribute such integrations, but I’ve been thinking for a while that chatbots as is, are cool and all, but to actually draw value from machine learning, an LLM should only be relevant to generate machine to human descriptions.

I am looking forward to AI running in the background, looking at what you’re doing and kicking in with a popup when it “has an idea” or detects an error that it knows how to fix. I would love to see this functionality in an IDE.

Example:
I’m writing a rather simply class that contains a few methods. The AI understands what I’m trying to do, but isn’t sure so it asks me what my goal with the current method is. I type in a quick sentence and it proposes me to complete the method, even offers me multiple options. But that would require a loooot of context it could survey, or access to a repository, or both.

I do like the little things like Notions AI helper. It’s rather simple but also pretty quick. When I’m working on projects, it keeps a summary for each project ready, which task is next, by whom it has to be done, it saves me time to go into the project and check each and every task. Really nifty! I want more things like that.

1 Like

Thanks for your response! Great perspective. Let’s see how it evolves and maybe even how we can built solutions like this. Groq’s fast inference speeds open up whole new solutions. I came across a few people creating voice assistants nearly responding in real time.

1 Like

I recall that groq is using LPUs / their own designed hardware to run their LLM, is this correct? To assume the Model itself would be much faster would thus be incorrect, I assume?

I really don’t know any details, but if that’s the case we would have to wait for LPUs becoming more available.

Yes they call their ‘Groq chip’ a LPU but i believe it runs Mixtral 8x7B-32k (an opensource model) So it’s really about the hardware being way faster at running these models.

The thing with Groq’s LPUs is that they have really low (RAM memory? idk. the exact details) I believe you need around 600 of these chips to run the Mixtral model. So these exact chips will probably be only available to larger company’s and by making API calls. But there will probably be many more developments in these kinds of hardware soon.

Ah yes, I have been saying this for a long time.

If you think about it, it’s silly how we scale everything horizontally these days. Kubernetes, AWS, Azure Virtual Appliances - and yet we still run the same old principle under the hood.

Imagine squeezing more and more cores and performance into a CPU to the extent where we have to cool them with water running through our electronics instead of finding ways to evenly distribute the calculation over multiple smaller CPUs, thus dividing the load and energy consumption. We could even push this to the extent where not used cores would completely shutdown.

I think Groq is aware that the time of “classic” hardware is running thin. Woulnd’t be surprised if this soon finds its way into consumer devices as well.

Anyhow, we have come close to hit a wall with this approach and that we now see variations popping up is a good sign. I for one am very psyched about this!

For what I’ve seen it’s the CPU Cache that needed to be significantly larger for an LPU to provide such an improvement, but I do not understand enough about it either.

But my hope for consumers lie in tiny AIs that have very specific tasks at which they exceed, requiring not more than a raspberry pi or even a small chip that could be plugged in through USB. I think I have seen AI “Accelerators” somewhere … Kind of exciting to think we could plug in a tiny “person” that helps us with work, even if it’s still very simple.

Totally agree that a little ‘intelligence’ everywhere could be really interesting. Can’t wait to tinker with local models with interesting abilities able to run on small energy efficient devices.

I saw some things about a whole new paradigms in computing. Quantum? thermodynamic? Non digital?(I can’t go into detail) Guillaume Verdon (Beff Jezos), a recent guest at the Lex Fridman Podcast has a company researching new kinds of chips. Extropic.ai

Here they tell about their efforts:

The human brain runs extremely energy efficient. There must be interesting new computing paradigms we will see in our lifetime. (In 10 minutes I turn 27)

2 Likes

I just watched Wes’ newest video and I am just happy clapping here, Blackwell seems to take that horizontal scaling approach that I just described before hahaha, that’s so great!

We’re moving in the right direction, more and more, and quickly!

Extropics work is very intersting as well, I think Sam Altman is / was interested for a good reason. The analog approach is, while I don’t understand most of what they said, pretty exciting.

1 Like