Get the top HN stories in your inbox every day.
fbrncci
theturtletalks
Won’t ChatGPT eventually eat your lunch? Once ChatGPT allows uploading documents (embeddings), what good will your app be?
The tools you’re talking about like math, wiki, or search are already built as plugins on ChatGPT.
I see so many AI apps being built, but I think ChatGPT will be general enough to cover 85-90% of use-cases using the chat UI.
chaxor
The most important aspect of langchain is NOT using OpenAI for the LM.
The most useful aspect of using langchain is to use it with Galpaca (or vicuna/koala/etc) to spin up an assistant for your home.
This way, you can push all of your files through it - even petabytes or terabytes of files, at a fraction of the cost - and have it organize things for you. No privacy problems, no extreme costs, just ease of use, low latency, offline, blazingly fast beauty. That's at least the trajectory.
Meta may soon release an improvement to Galactica similar to Galpaca (GeorgiaTech attempt) more officially (perhaps with more multimodal focus), which will likely improve upon the llama based models even further.
ChatGPT is just one model among many here, and it's not even the first to use RLHF (Deepmind, as usual, was a bit earlier).
The simple task of downloading Redpamajas/thePile/etc and getting a vector db for it locally, and enhancing it with local files effectively brings a local Google to everyone, and it may only require a decent spinning disk HD for the DB storage with the typical langchain LLM setup to have a completely local 'jarvis'-like assistant. (Sure, I know some people care about 'news'-like info that requires connectivity, but most things don't)
0xDEF
The vast majority of people building LM apps with (or without) LangChain are using OpenAI.
I sincerely hope local LM tech like Galpaca (or vicuna/koala/etc) succeed but I don't understand why we are collectively pretending they are currently anywhere near gpt-3.5-turbo both in terms of speed and quality. Honestly the local models feel more like first generation BERT/GPT-1 models that have been fine-tuned for QA using RLHF.
fbrncci
It's more or less an advanced personal project to stay on top of the LLM learning curve, rather than just being exposed to news and press releases. I also have an appetite for further wrapping my mind around all of this. I already work in the AI space as web-developer on the B2B & enterprise side of things. My opinion here is that there are going to be loads of use-cases and necessary plugins, which for privacy, legal and security reasons need a proprietary solution and won't be able to interface with any third party APIs, plugins or frameworks.
theturtletalks
I completely agree with you on AI being extensively integrated into existing apps and being leveraged that way.
But most of the AI apps I see are just like a “skin” on ChatGPT API.
I do think there is value in a universal chat UI that can connect to GPT-3 and other models.
hospitalJail
We cannot give my company's information to OpenAI/MS. No legal paperwork will change this. This information is so important, it is only on offline computers.
kozikow
I've experimented with LangChain for my chatbot as well, but ultimately, I resorted to using custom Python. Here are a few issues I faced with LangChain:
- By developing your own solutions, you can engineer specific components that would be provided by LangChain to better suit your use case. For example, by fine-tuning to your use case you can have better results with converation history, context and summarization better by prompt engineering. If you look at prompts within langchain they are pretty basic.
- LangChain is designed around the idea that an entire chat logic resides within a single "REPL loop." In my use case, I had a single-page web app frontend, a standard web "RESTful" backend, and a separate chat service. Different parts of the information are stored and managed by these components. Using LangChain would have forced me to consolidate all logic into the chat service, which doesn't align with the overall architecture of my system beyond just the chat functionality.
Please note that I'm not a LangChain expert, so my assessment might not be entirely accurate about its capabilities. However, based on my evaluation, LangChain introduced too many constraints in comparison to what it provided.
msikora
This, I just pretty much ended up using the basic LLMChain and do my own custom flow. The built in agents are close to useless for anything but a toy project; it is simply way too unreliable.
danvass
We've been building LLM chains for over a year and found that whilst for simple use-cases it's easy enough to grab a couple of APIs. However, having a Notebook experience built for iterating and collaborating whilst managing the complexity is something we've seen companies care deeply about.
nbardy
I’m not convinced yet, but if they can provide nice templating and reusability.
There is plenty room for code reuse in prompting.
rcme
LangChain has been so frequently discussed that I thought it must be this amazing piece of software. I was recently reading about vector databases and how they can be used to provide context to LLMs. I came across a LangChain class called RetrievalQA, which takes in a vector database and a question and produces and answer based on documents stored in the vector db. My curiosity was piqued! How did it work? Well... it works like this:
prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:"""
nextworddev
It got a lot of traction on non-coders thinking it’s doing some magic, but if you read the source it boils down to some brittle prompts and lots of Python class boilerplate.
There’s some useful parts though
rcme
Just out of curiosity, what are the useful parts?
nextworddev
It’s good for building vector indices without worrying about writing adapters to milvus, pinecone, qdrant etc separately - in case you want to switch out later
19h
The class structure provides a clean API for building on, even if the internals are basic. With some refinement, it could be a good starting point for more advanced models.
ryanjshaw
I had the same experience and kept wondering if I was missing something important. I'm not a fan of Python, so I was anxious about not using the thing everybody recommended, but for my project I ultimately went with what I know well (C#). I've happily had zero issues.
LangChain docs and tutorials were useful for understanding the popular practices for approaching AI-driven development, but the biggest challenge by far has been getting a baseline prompt and measuring performance of alternative implementations against that in a sensible way that doesn't break the bank. Mitchell Hashimotos's Prompt Engineering article [1] was way more helpful in this regard than anything I saw in LangChain.
To that end I've also been working on a tool to save me money by caching requests and responses, blocking unexpectedly expensive requests, keeping a granular history of requests for prompt cost analysis, etc. Maybe I should open source it and get some VC bux too?
[1] https://mitchellh.com/writing/prompt-engineering-vs-blind-pr...
fbrncci
But wait, there is more! In Langchain you can build constitutional chains ontop of your chains, to validate if the answer was really helpful, by doing just one more API call, with a new prompt, asking if the answer answered the question based on the initial prompt, in an helpful way! And if it didn't, revise the answer with another API call to be more helpful! And then you can chain these chains with even further API calls until you went through as much prompts you think are necessary to answer a single sentence question (What is the weather today on the moon?).
fzliu
LangChain is meant to reduce/remove the amount of boilerplate code needed to build a lot of applications with LLMs. I see your point, but I still think LangChain is useful for a particular segment of early stage developers.
theptrk
I tried to make a youtube video exploring the code and it was fairly short https://www.youtube.com/watch?v=Joby-58DuBE. I think if the prompts were put front and center in the documentation it would be clear up a lot of mystery.
Karrot_Kream
The default "document splitter" just splits along double newlines. It took me aback when I realized they made an entire class around this.
convexfunction
I'm not sure if "helpful answer" as opposed to "answer" makes much of a difference in answer quality -- I'd believe it helps a little, just don't know that it's been studied -- but a lot of silly stuff like that does definitely make a big difference in response quality on certain tasks. "Let's think step by step:" at the end of your prompt is probably the best-known one: https://arxiv.org/pdf/2205.11916.pdf
m3kw9
It basically gathers text that are similar to what you are asking and feed it into the prompt, yes. No magic. The worse part is that if you ask “please get me the summary to this doc” it will actually search the vector db using the entire question. It’s not very smart. Depending on how you split the embedding you could end up with a bunch of crap
artathacanadian
We’re building an easier-to-use langchain that lets you preprocess inputs and remove unnecessary wrapper text by going like “please get me the summary to this {{INPUT}}”
loveparade
Am I the only one who is not convinced by the value proposition of langchain? 99% of it are interface definitions and implementations for external tools, most of which are super straightforward. I can write integrations for what my app needs in less than an hour myself, why bring in a heavily opinionated external framework? It kind of feels like the npm "left-pad" to me. Everyone just uses it because it seems popular, not because they need it.
crazyedgar
For us LangChain actually caused more problems than it solved. We had a system in production which after working fine a few weeks suddenly started experiencing frequent failures (more than 30% of requests). On digging it seems that LangChain sets a default timeout of 60 seconds for every requests. And this behaviour isn't documented! Such spurious decisions made by LangChain are everywhere, and will all eventually come back to bite. In the end we replaced everything with vanilla request clients. Definitely not recommended to build a system on a library that provides very limited value while hiding a huge amount of details and decisions from you.
Spivak
Langchain is absolutely perfect though, it's bad enough that you'll be driven to write something better out of pure frustration but gives you enough good ideas and breadcrumbs to actually do it.
It's probably the best on-ramp for "practical uses of llms" because it scratches just the right developer itch.
ErikBjare
This is the big take-away
bfg_damien
I am slowly coming around to the same conclusion. It isn't always clear how some agent types are different from others. Sometimes the prompts expect JSON blobs and sometimes they expect something else. I tried it out because I could see the potential, but I dont think it's architect-ed in a way that is suitable for things beyond simple PoCs.
It would probably be much better to start with the basic OpenAI API and then build on top of it.
What I find particularly frustrating is the difficulty in easily interfacing with my existing python tools (not like add two numbers, but somewhat complex analytics on top of structured data). If anybody has any success with interfacing with existing tools/scripts, would love to know how people are going about doing it.
newswasboring
It's brilliant for experimentation and prototyping though. Granted I've not deployed anything llm related yet so I have not thought about it yet, but I don't want to just start writing every integration I think I need by hand just to experiment with it.
gumby
> I can write integrations for what my app needs in less than an hour myself,
Or just ask ChatGPT to do it...
Joking aside, I think 'npm "left-pad"' describes it perfectly.
cube2222
Yeah, the basics of LangChain are fairly simple, and reimplementing a loop like that in Go, including tool usage, was very straightforward when I was writing Cuttlefish[0] (a toy desktop chat app for ChatGPT that can use stuff like your local terminal or Google).
The magic in LangChain, though, is the ecosystem. I.e. they have integrations with tons of indexes, they have many tool implementations, etc. This is the real value of LangChain. The core ReAct loop is quite trivial (as this article demonstrates).
adityapurwa
I got the chance to try Langchain as part of a hiring process. I was already having my eye on it for a personal projects though.
The moment I tried it and went through the docs, the entire abstraction feels weird for me. I know a bit here and there about LLM, but Langchain make me feels like Im learning something entirely new.
How agent and tools work and how to write one wasnt straightforward from the docs, and the idea of having an AI attach itself to an eval or writing its own error/hallucination-prone API request based on a docs doesnt give me a lot of confidence.
The hiring assignment specifically mentioned to use Langchain thought, so I did. But just as a glorified abstraction to call GPT and parses the NL output as JSON.
I did the actual API call, post-processing, etc. manually. Which I have granular control over it. Also cheaper in terms of token usages. You could say I ended writing my own agent/tool that doesnt exactly match Langchain specifications but it works.
I guess Langchain had its use case. But it feels pretty weird to use for me.
minimaxir
LangChain and the ReAct paper that helped codify the implementation are both less than a year old.
A hiring assignment suggesting it is…weird.
adityapurwa
Its for a company that helps researchers, so I guess that’s why.
triyambakam
What type of position was the assessment for?
adityapurwa
It was a senior full stack position for a company that build products for researchers.
lxe
I've been working with langchain and llamaindex and did notice that it's a pretty hefty abstraction on top of pretty simple concepts and I also eventually ended up dropping both and simply write the underlying code without the framework on top.
zomglings
Ditto. But with faiss and openai api.
undefined
KevinBenSmith
Pretty much sums up my experience as well.
okhat
There’s always DSP for those who need a lightweight but powerful programming model — not a library of predefined prompts and integrations.
It’s a very different experience from the hand-holding of LangChain, but it packs reusable magic in generic constructs like annotate, compile, etc that work with arbitrary programs.
barking_biscuit
Didn't know about this. Looks promising!
ukuina
I cannot praise Deepset Haystack enough for how simple they make things compared to LangChain, between the Preprocessor, the Reader/Retriever, and the PromptNode - the APIs, docs, and tutorials are quite easy to modify to your use-case.
Not affiliated, just a happy defector from LangChain.
iamflimflam1
Is it open source? Wasn’t clear from the website.
aantti
Yes, Haystack is Apache 2.0 :) https://haystack.deepset.ai
saulpw
I also was underwhelmed by langchain, and started implementing my own "AIPL" (Array-Inspired Pipeline Language) which turns these "chains" into straightforward, linear scripts. It's very early days but already it feels like the right direction for experimenting with this stuff. (I'm looking for collaborators if anyone is interested!)
KevinBenSmith
As someone who has created several LLM-based applications running in production, my personal experience with langchain has been that it is too high of an abstraction for steps that in the end are actually fairly simple.
And as soon as you want to slightly modify something to better accomodate your use-case, you are trapped in layers & layers of Python boiler plate code and unnecessary abstractions.
Maybe our llm applications haven’t been complex enough to warrent the use of langchain, but if that’s the case, then I wonder how many of such complex applications actually exist today.
-> Anyways, I came away feeling quite let down by the hype.
For my own personal workflow, a more “hackable” architecture would be much more valuable. Totally fine if that means it’s less “general”. As a comparison, I remember the early days of HugginfaceTransformers where they did not try to create a 100% high-level general abstraction on top of every conceivable Neural Network architecture. Instead, each model architecture was somewhat separate from one another, making it much easier to “hack” it.
dpbrinkm
>someone who has created several LLM-based applications running in production
Come and talk about what you are doing and challenges of it at our LLM in production virtual conference? https://home.mlops.community/home/events/llm-in-prod-part-ii...
19h
Comparing Langchain to Hugging Face Transformers is apples and oranges. One is for research, one is for production. Production ML requires more abstraction, not less.
crazyedgar
I disagree. Production systems don't need to be full of AbstractSingletonProxyFactoryBeans which is basically what LangChain is. For example, Linux certainly isn't.
zyang
I'm glad I wasn't the only one that felt Langchain had a ton of redundant abstractions engineered to gain clout for vc money. Here is an example:
AnalyzeDocumentChain[1] just wraps RecursiveCharacterTextSplitter[2]. It serves no real purpose except padding the api doc.
[1] https://js.langchain.com/docs/modules/chains/other_chains/an... [2] https://js.langchain.com/docs/modules/chains/other_chains/su...
crazyedgar
It's worse than that. The documentation is a confusing mess that completely omits the explanation of key default parameters and details. And the abstractions are horrendously brittle. And difficult to fix, because there are too many layers.
The best use of LangChain is probably just looking at the included prompts in the source code for inspiration.
convexfunction
If you know little about prompt engineering and want to throw together a demo of something that kind of works extremely quickly, or experiment with an LLM agent exactly as it's defined in some paper, LangChain is pretty useful.
If you want to develop a real LLM application, you're probably better off skipping the library completely, or at least fully understand each abstraction to make sure it does everything you want before you decide you want to incorporate it.
Get the top HN stories in your inbox every day.
I work with Langchain on a daily basis now, and so often I find myself asking; do I really need a whole LLM framework for this? At this point, the assistant I am writing, will likely be more stable rewritten in pure Python. The deeper and more complex the application becomes, the more of a risk Langchain seems to become to keeping it maintainable. But even at less complex levels, if I want to do this:
1. Have a huge dataset of documents.
2. Want to ask questions and have an LLM chat conversation based on these documents.
3. Be able to implement tools like math, wiki or Google search on top of the retrieval.
4. Implement memory management for longer conversations.
Its still a lot more straightforward to maintain it in Python. The only thing where it becomes interesting is having agents execute async, which is not that easy replicate, but at the moment agents are not that helpful. Not trying to diss Langchain too much here, because its such an awesome framework, but I can't help seeing past it other than just being a helpful tool to understand LLM's and LLM programming for now.