Upgrade Hardware or Cloud Compute Services?

Hi All
Currently I am considering to buy a refurbished HPE ProLiant DL380 G9 to fine tune my own LLM model at home. I was going to deep compare it to cloud computing costs but I made 3 API calls on amazon AWS and that used up $300 in free credits , I also like hardware and owning it. I just think maybe AI development is so fast and for that reason it could be smarter to not buy end of life hardware , any input appreciated!

I agree with you twice. The development is that fast. The question you raised is valid.
On the other side I started last year with a smaller hardware and I learned so much - just by experimentig with local models. Tending also to HW.
But since yesterday the local model size seems to raise up to 370GB - that is quite a bit to handle with own hardware.

1 Like

I vote for owning the hardware, if you accidentally invent Skynet you can smash it with a hammer instead of having it bloom in a big server farm.

Just kidding (?) but srsly I do think a fixed cost will make you more free to experiment and be creative, thus learning quicker.

2 Likes

Have you considered renting a root server? Saves you a lot of trouble with storing the hardware, power consumption, maintenance, etc.

I’m running my stuff on a hetzner root with 16 cores and 128GB RAM, but I’m also using it for a lof of different things on the side (mysql database, discord bot, game server) Still more than enough juice left for all sorts of experiments with all things AI. Costs me roughly 120€/month which is actually fair for what I’m squeezing out of it, hehe

Just had a look at the post Flabbergasted!.. from @dogin20 (thanks for sharing the link to the videos) Aera of one bit LLMs
That give a bit of hope for the own hardware approach :wink:

What about privacy and proprietary… ness? And. Wouldn’t it be nice to not be reliant on an internet connection… ya know… in case, the worst should come to pass or the corporate overlords decide to ratio 1’s and 0’s. At least you could still write your own books and render movies. …maybe have a digital friend to share the end of the world with?? :blush: