Conducting deep web searches and gathering sources is one of the main things I’ve been using LLMs for. How far away are we from being able to self-host something like Claude’s web search capabilities? Or even just a service where I’d pay with my money instead of my data?
You must log in or register to comment.
2 monthsPerplexica now called Vane is an open source version of perplexity. It lets you search the web with an llm that cites sources for its claims and can be selfhosted https://github.com/BunsDev/perplexica-search-engine-ai
- 2 months
Probably not far? I’d look into some LLM that can run commands with API, hook it to some web search API, and then work on the results.
- 2 months
Openwebui+searxng on a AMD strix board.
Pro: works like a charm, low power consumption, fast, “big” , LLM (running qwen3.6 35B A3B + gemma4 E4B for website summaries and other smaller tasks)
Con: strix boards start at 2k€, more in USA because of tarrifs
- 2 months
Curious why do you swap between Qwen and E4B. On my hardware they perform with similar tps. Qwen 3.6 35B spits out 80-100tps on AMD 9700 and E4B gives me about the same tps.
- 2 months
To avoid context switching on the GPU. OpenWebUi for example uses it for memory and title generation.
Those are not performance critical and background tasks, so instead of slowing down qwen, we just outsource this stuff to the NPU.
- 2 months
Oh I see. Okay this makes sense. I just throw Qwen 3.6 35B Q8 on 2 GPUs and use it for everything but coding agent.
- 2 months
For those who want to know more, rough setup:
- llama-cpp rocmfp4 fork
- currently custom quantized qwen3.6 35B A3B model, working on publishing
- be3 embedding and reranker, also GPU
- gemma4-e4b via FastFlowLM on NPU!
- OpenWebUI and searxng as docker containers on a Pi currently
We get 70-100tok/s generation. Four slots with 256k context length each.
We use a smaller Board with “only” 64GB of shared LPDDR5X. Bottleneck is memory speed, rocmfp4 quants help a lot.
As soon as I get my imatrix calibration right, I will publish the quantized versions.
Most existing quantized models are broken. The authors did some not supported stuff (like using a already quantized model and requantize it) that you may get issues with coherence or sudden Chinese words in the output.
That is not an issue with rocmfp4 but with vibe coders and agent psychosis.
- 2 months
Do you have a walk through for setup?
I’m on the strix halo 128 gb variant and while I got ollama working fine, i haven’t gotten any of these multi headed setups working
- 2 months
I am on Gentoo for it, but everything with a decent rocm should work.
Have a look for llama-swap, that handles multi head endpoints.
Also, as you are on a big board, you can quantize yourself, as the BF16 version of qwen has only 72gb.
I will try and post a full writeup next days. But feel free to dm me, if you need some guidance on quantize or more.
I am using this fork currently: https://github.com/charlie12345/ROCmFPX
Stuff happens fast currently, so may be worth to wait a week or two ig you need something super stable, but if you are up for experimenting, that’s the way to go
- 2 months
THis is great, thanks. I’m on the z-13 and needed to use it for a work project, which is wrapping up soon. I’m planning on re-building it as a locally hosted agent support machine.
I do this today.
Reasonable options are the Brave Search API (free/paid) and SearXNG (self hosted).
Then you need a harness like Opencode or Pi Agent that can use these search engines.
- 2 months
If you have the equipment and about 5TB of storage Fable 5 AI is available by Torrent on Pirate Bay right now. :)
It was just a meme there is no download available. Itd be much bigger news if it actually leaked
- sem@piefed.blahaj.zoneEnglish2 months
What’s an example of something you can do with this? I’m not familiar with Claude
- 2 months
“Hey Claude, research for me current research to Nuclear Fusion. What are the biggest hurdles what are the next steps, and how promising is private research” enabling the research feature will give you a report, Fact checked (not clean but ok ish), and all the sources for it.
Claude will spin up a bunch of workers and search the web, following leads, and so on.
One of the few actual useful features of AI IMHO
- 2 months
I know you said not clean, but Claude and other LLMs with this ability can and do still hallucinate even when researching. You can have it give you direct source links (not just the attributes that it gives by default) with explicit instructions to quote the exact finding, word for word, and sometimes it will suddenly tell you that it was made up and not actually in the attributed page. Not every time, but it’s something to be very careful with.
Google’s own LLM struggles with this issue as well.
