Get the top HN stories in your inbox every day.
Nican
survirtual
I’ve dived deep into IPFS and built several prototypes on top of it. It ended up not being performant enough for me, and that was after heavily modifying the codebase so that it was true p2p browser & server (their webrtc transport had a lot of issues and they didn’t seem too interested in it, but my needs required it as a backbone).
The security was also a concern, and the scaling had issues. Pinning millions of small items got so slow it would not function. Then I ended up having concerns over hashed based addressing being easy to censor with the architecture IPFS was using (more hub & spoke than anything, given signaling and relay servers were centralized).
I could go on but I ran into so many issues I ended up implementing my own solution that did everything I wanted. Wanted to squeeze even more performance, I’ve been converting that solution to Rust.
This was a couple years ago so maybe things have changed since I used it. Last I checked, they seemed busy on Filecoin.
The idea of IPFS is great and I want to see it succeed, but I think that they got so caught up in their jargon and modularity, the project lost track of some fundamentals.
oldgregg
Honestly I think Consensys/IPFS/Libp2p is just some corporatized way to derail real P2P and decentralization. Their libraries are total garbage. Lots of complicated code that simply doesn't work. No documentation. I mean look how much IPFS and Libp2p is pumped but IT DOESN'T WORK. IPNS is a joke. All way overengineered crap that does everything but actually nothing. Look at the $$$ and pedigree behind Consensys it's 100% establishment.
survirtual
Anything is possible, but having some limited github interactions with the core team, that seems unlikely. My impression is that they are a passionate group that hit the jackpot a bit prematurely.
Their intentions seemed good to me they just have an ungodly amount of financing while perhaps lacking a core vision & understanding of what is at stake.
Put another way, seems more like an academic building something rather than a seasoned industry pro.
In any case, I don’t want to disparage their project. I learned a lot from their code & concepts. We are all on our own roads towards brilliance, contributing to each other in all kinds of ways.
1vuio0pswjnm7
Peer-to-peer is too general and too useful to be spoilt by "blockchain". But there is so much money behind this crap it is a formidable virus that can infect any project.
ConsenSys is a company formed by Ethereum co-founder. He has sold out to the big banks, in the opinion of some Consensys shareholders.
https://finance.yahoo.com/news/consensys-shareholders-readyi...
"Falls says he believes both MetaMask and Infura - what he calls "the crown jewels" of ConsenSys - could have been decentralized and tokenized, and that their projected use is "completely anathema" to the peer-to-peer principles of the space.
Meanwhile, a number of teams are looking to bring "institutional DeFi" - with its known counterparties and compliant custody arrangements - to the marketplace.
"Forget about the shareholders for a minute," Falls said. "Think about the consequences of the change in the influence over these infrastructure pieces.""
Out of curiosity I was looking at the evolution of IPFS hashes. They have gotten more complex. IPFS now uses base58btc exclusively. That's "btc" as in Bitcoin. Something like rhash, which has traditionally supported hashes used in peer-to-peer protocols, has no support for base58btc.
People are now trying to associate IPFS with "Web3". For example, check out this paper published a couple of weeks ago.
"Studying the workload of a fully decentralised Web3 system: IPFS"
mhluongo
You're confusing ConsenSys and Protocol Labs.
ConsenSys was founded by Joe Lubin, and is the corporate entity behind MetaMask, Infura, etc. It counts MasterCard and JP Morgan as investors.
Protocol Labs was founded by Juan Benet and is the primary sponsor of IPFS and the dev team behind Filecoin. They're funded primarily from the private token sales of FIL.
Agree with the sibling comment that the focus on modularity and solving "all of the things" rather than focusing on adoption seems to be a pattern with Protocol Labs, though I'm a big fan of what they're trying to build.
KirillPanov
Same experience here.
The #1 lesson they failed to learn from Bittorrent and Tox: your DHT must be sessionless.
QUIC is not a solution to this particular problem. Although QUIC uses UDP, it still creates and manages sessions. This is incompatible with decentralized scaling. At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.
2color
IPFS encompasses more than just a DHT: - representing content addressed data with hashes in CIDs - Discovering them (via the DHT) - Transfering (Bitswap, HTTP, Graphsync, sneakernet and more in development)
I'm not 100% certain what you mean by the DHT being sessionless.
> At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.
You don't have to route content via the public DHT. Depending on what kind of data you're publishing, you can use a private DHT swarm, or delegated routing with HTTP.
> This is incompatible with decentralized scaling.
anacrolix
Absolutely. Too many people try to implement DHT on top of stateful connections, it doesn't scale. It must be as cheap as a single packet or two, with no context.
password4321
> my own solution that did everything I wanted
Your ideas are intriguing to me and I wish to subscribe to your newsletter!
(I really would appreciate any pointers to additional info on something that actually works...)
survirtual
If you find something, let me know. It would save me a lot of trouble.
Until then, what I’m building unfortunately doesn’t seem to have a peer. So I will keep at it. If it gets to a point where I believe it can be useful to others, I will share it freely.
klabb3
I’m building a connectivity lib in Golang for relay-assisted p2p over tcp. The implementation is simple, uses only a single network connection when idle, and establishes connections very quickly and reliably. Downside is it’s not decentralized and it’s also not UDP, in case that’s your thing. Oh and BYO auth.
Please reach out if it fits your use case.
chezball
@survirtual, would love to see what you used your own implementation for. Is it open source? Or a product we might be interested in?
ComputerGuru
ZFS (Zettabyte File System) is very much separate from Z-Library (and its alternative frontends like libgen).
Nican
Thanks. My brain crossed wires between Z-Library and IPFS. I updated my post.
ComputerGuru
Completely understandable!
benneh
IPFS is not the only option available. Swarm's been making a lot of progress.
Hackernoon released a blog post comparing the two projects.
https://hackernoon.com/whats-the-difference-between-ipfs-and...
steponlego
Works great, that's old news which is no longer current.
nestorD
I just did a quick test, it could do with slightly fuzzier searching ('Epub' as an extension got me no result while 'epub' did, a drop down menu with options might be better/simpler for some fields) but, otherwise, it seems functional and useful.
Cort3z
It seems case sensitive on some (or all?) of the input fields. At least for the author field in some of my searches.
jraby3
I got an error when visiting zlib.zu1k.com/. Error code: 1020 Ray ID: 7819e5389c898e4e Country: IL Data center: tlv03 IP: 85.65.187.66 Timestamp: 2022-12-30 09:54:40 UTC
0ct4via
It's possible they were "hugged" enough to instigate a firewall rule dramatically cutting down allowed visitors, which would explain the Error 1020.
Might be worth reaching out to the site's creator -- info taken from their site [https://zu1k-com/about/]:
"If you have any questions to communicate with me, the preferred way is to comment directly in the comment area of the corresponding blog post, and I will receive an email notification
In addition, you can also reach me in the following ways:
Email: i@zu1k.com
Twitter: @zu1k_lv
GitHub: @zu1k
Keybase: @zu1k
Matrix: @zu1k:mozilla.org
My PGP public key can be obtained at https://pgp.zu1k.com or hkps: //keyserver.ubuntu.com
( 2A65 F6F3 1EDA D922 D7E7 E97B AE38 1A8F B1EF 2CC8)"
neop1x
People should stop over-using CloudFlare for everything. IMO it is not good for IFS "hosting", either.
They act like gatekeepers. Along with blocking, as in this case, they make browser assumptions that force visitors to use one of the most recent mainstream "approved" and unmodified browsers.
For other browsers, they serve "Checking if the site connection is secure" with javascript fingerprinting and a captcha, and sometimes even solving the captcha won't let me in. i have also seen frontends misbehaving due to them not expecting cloudflare error response codes.
Some dvelopers apparenly just blindly enable cloudflare and don't care. :(
SanjayMehta
Same here. Was working a few hours ago.
deanc
Is the catalog on IPFS different to the catalog available via one of the libgen front-ends? I just performed a search on both for one title and it found it right away on a libgen front-end, but not here.
DoItToMe81
Z-library began as a Libgen mirror but split off and stopped sharing uploads. It's kind of scummy, but I guess that just means people need to put more effort into uploading its contents back to Libgen.
satvikpendem
> libgen front-ends
Libgen is not the same as Z-library, there are many books on Z that are not available on libgen.
Thorentis
Where does "Z" exist as an entity, and how does one add books to it, or figure out what books belong to Z or not?
derrida
I had the opposite... a book I was looking for on a lib gen front end a few days ago but could not find came up here (the Narada translation of Dhammapada if curious)
undefined
jerojero
Something that has always bothered me about z-lib and libgen and so on is the impossibility of being able to tell quality from the different uploads.
Sometimes the epubs are very good, with a good cover and metadata as well as chapters and no weird aberrations; other times, they're unreadable. But the only thing you have to judge quality is size and general description of the upload which in this website it's been reduced so everything looks the same.
I think the website looks good, but yeah, would be nice if we could rate quality in some way.
ok123456
Large pdf and djvu files generally means low quality scans.
akiselev
I think that really depends on the type of book. I've found the large PDFs to be by far the most reliable for books printed before the 20th century because - save for Google Books - no one's OCR is tuned for the kind of printing errors, age related wear and tear, and fonts used over the centuries.
Manuscripts (which are admittedly very niche) often have to be high resolution scans or photographs just to be readable, but zlib and libgen don't really have many of those.
zozbot234
Surely books published before the 20th century are in the public domain by now, though?
dredmorbius
That depends, variously on a number of factors.
A straight-text book can run as little as 500--750 KiB for 100--200 pages or so.
ePubs may run smaller as they're a compressed archive of HTML files, so how large the straight text is is largely a matter of the additional markup and stylesheets applied. I've generated PDFs from Markdown which are fairly comparable.
Most mainstream trade press books run about 3--6 MiB as PDFs, if they have few images or graphics.
Books with a heavy graphics content can easily swell to 30--300 MiB.
And scanned rather than generated PDFs tend to run similarly sized. Many of those scans are, however, excellent quality.
The largest ebook within my own collection for quite some time was a copy of Lyell's Geography, downloaded from the Internet Archive and based on a library scan of a 19th-century printing. On a colour tablet, it's actually pretty nice reading, on a B&W E-ink tablet, such scans often have a significant background ghosting which may or may not be eliminated by contrast adjustments in your reader.
For newer materials, I've seen any number of issues with ebooks:
- Clean single-page scans with OCR'd text for highlighting and copy/paste are generally my favourite. There's something about original print layouts that I still find preferable to ePubs and other digital-native formats.
- Native PDF can also be quite good though they're far more prone to designeritis, where someone has thought they could improve over ink-on-paper conventions. If you think you can do this, you are almost certainly wrong.
- ePubs rerendered as PDFs are among the worst options. I prefer PDFs generally, but not in this case.
- ePubs ... can ... be reasonably good, though again the principle problem is excessive flexibility for designers. Less is more. I really miss having a single consistent layout of the text, rather than something which reflows as font faces, sizes, and spacing are changed. As a saving grace, cringeworthy (and migrane-inducing) font choices can be overridden with The One True Serif Font.
- Various low-quality scans. 2-up, lots of page skew and placement variation, heavily-marked texts, and such. I'll still almost always prefer these to an ePub or other digital-native format, though the reading can be much harder.
- Tiny-font 3-column scientific publications. Even on my 13" e-ink reader, these can be a challenge, and I'll occasionally resort to a sub-page rendering (this is natively supported in the Onyx NeoReader and several other e-book readers). Why such publications continue to insist on this format as we approach the 2nd quarter of the 21st century I've no idea at all. Scans based on older journal pubs (from ~1950s -- 1980s or so) with both physical wear and scan artefacts can be especially challenging.
LarryMullins
The books are free, if you download the wrong version of a book you can simply try another and it costs you no more than a minute of your time.
Do you know how it worked before we had the internet for books? You had to go to a library yourself, which may have taken an hour of your day just in the journey, then hope the library had the book you wanted. If not, you may have to wait weeks for another library in the system to mail the book to your library.
jerojero
I'm describing a problem, the fact that the books are free and that this problem causes a minor inconvenience doesn't make it not a problem. It's important to be aware of these things.
leokennis
One mitigating factor is that it’s very easy to “correct” all but the very worst ebooks in Calibre:
- Metadata and a cover are one click away
- Removing excessive spacing, padding etc. is a few clicks
Tams80
Pirates being picky, eh?
Don't get me wrong, I do appreciate the access that libgen and z-lib provide and think on the whole it is beneficial to society. It is still not paying for people's work though.
AlotOfReading
As a counterpoint, have you looked at your local library's ebook offerings? Publishers have created a situation where digital books are significantly more expensive than physical copies and expire far more quickly. We're also just coming out of a two year period where physical libraries effectively didn't exist and most library systems hadn't expanded their digital offerings to meet demand.
I still buy books, but these sites have largely replaced traditional libraries for my partner and I because they allow us to read what we actually want instead of Oprah's book club titles or the 50 digital self-help guides my local library stocks with their limited resources. I recognize that they're simply serving what most people want, but the alternative for me is simply not reading recreationally.
The other side of this conversation is that even if we did get rid of pirate libraries there would still be huge, unresolved issues with author compensation. There's been a dramatic decline in how much authors (and the other people directly involved) take home over the past couple decades [1] that has little to do with the marginal number of pirate readers.
[1] https://authorsguild.org/news/authors-guild-survey-shows-dra...
godelski
A big reason piracy exists is because the functionality of these systems is better than the official channels. There's a reason streaming services led to a dramatic decrease in piracy. There's also a reason that piracy is making a return. Yeah, pirates are picky and they push other services forward.
cratermoon
A question about z-lib, libgen, and regular libraries
Suppose I check an e-book out of my local library. Because Reasons, the library doesn't "own" unlimited "copies" of the book, so each "copy" can only be checked out to one patron at a time and if there are enough holds then a patron can't renew a checked out book. In short, ebooks in libraries are just like regular paper books, except to you don't have leave your house.
I had this ebook on hold for three months because it's very popular, but finally have it on my device, but I only have 3 weeks before I can no longer read it because of holds. I get about 2/3rds the way through before lock closes on the bits on my device. It's a very popular book, it will be 3 months before I can read it again.
I go online, use some service, find the exact same book, down to all but the locking bits, and download it, finish reading it a few days later, and then forget about it. I might delete it later if I need room.
Is that a crime? Have either the author or publisher lost money? On the one hand, I can say yes because if the libraries purchased more "copies", the waiting list wouldn't be so long, I might have been able to renew it and finish it. On the other hand, I wasn't going to buy the book myself, and the library has to balance budget and demand, so they probably wouldn't purchase additional "copies".
What ethical questions do authors and readers see here?
tgsovlerkhgsel
> Is that a crime?
In many countries it is at least a copyright violation, although it should be noted that downloading (not sharing while torrenting) is legal in some places.
> Have either the author or publisher lost money?
On average, probably yes, because some people would buy the book, some libraries would get more budget or allocate more of the budget to that book, and most importantly, if you did re-borrow it in 3 months it'd further expand the waitlist, perhaps making someone go "screw that, I'm not waiting 4 months" and buy it.
OTOH, if you took all the money being spent on entertainment (books movies etc.), collected that as a tax, distributed it to the entertainment industry based on what people actually consume, and in exchange made everything available to everyone, there would be no loss of revenue and everyone would benefit from much higher, friction-free levels of access.
That's part of what made Netflix work. It extracted ~$120/year of revenue and gave access to everything. Now, with all the competing services, they probably still don't extract more than one service's yearly fee over the same population (because few people have more than 1-2 services, and the ones that just gave up and went back to piracy probably make up for the ones that now pay for 2 instead of 1 service) but people now have much less access.
Without a massive "media tax" which would have practical issues that don't make it feasible or desirable, some friction on free access is necessary or nobody would pay for content, and at the same time that friction harms society by limiting access. Piracy puts an upper bound on how much friction can be added before people go the yarrr way.
wccrawford
My layman's understanding of copyright law says you didn't commit a crime, but whoever sent you the book did commit a crime. My understanding is that it's not illegal to download a book, it's illegal to distribute a book. (Or other copyrighted work.)
1123581321
No, it is illegal to download (make a copy of) the book. To be legal, you would have to merely view the book without making a copy. This is the operating principle behind all those video streaming sites. This isn't a comment on what OP should have done.
justinclift
Doesn't this vary according to the location of the participants?
eg in some countries participation in some parts of it (uploading/downloading/etc) is legal, whereas in other countries it can be the other way around.
alwayslikethis
Wouldn't the transmission over the network of the bits of the video count as an act of copying?
wccrawford
I stand corrected. Thanks!
juuular
You might find this interesting:
Reverse engineering yet another ebook format (Nemanja Mijailovic) https://mijailovic.net/2022/12/25/hkpropel/
cratermoon
I did read that. The format he had wasn't really locked down, it was just a regular epub behind a crappy web-based viewer. The library epub downloads really do have DRM locks.
mdaniel
With discussion: https://news.ycombinator.com/item?id=34125958
vouaobrasil
I think the central ethical question to piracy is that ultimately, buying a book someone wrote is fair to the author. The author is part of a society -- our planet, did something to contribute to society, and it's fair that we reward that person if we also benefit from their work. Therefore, whenever you benefit from someone else's work, how do we reward them.
Now, I think this question is nuanced. For example, let's take Harry Potter. That book has sold millions. In my mind, J.K. Rowling has already been rewarded by humanity for her work many times over, so if you were to pirate her book then I think that is a far lesser evil than pirating a book written by an author who has barely made any sales.
Basically, I view the question of piracy as follows: (a) have we rewarded the author enough for that specific work? and (b) if not, then it's not right to pirate their book, and I should buy it. I think everyone should ask themselves that question.
The same with movies. Marvel movies are a money-making machine and they have more than enough money to keep them going. An indie film much less so.
The question of whether an author has been rewarded enough is tricky. For example, if Elon Musk wrote a book, I would have no problem pirating it in a microsecond because society rewarded him enough. But maybe that's not so clear-cut because the editor might be poor. Thus, I think it's safer to go on the basis of sales rather than the wealth of the author, except of course if the author is self-published. In that case, if they are rich, then who cares. Of course, what is rich? Well it's hard to give exact numbers but if they have 10 million dollars that's definitely rich enough.
kleinishere
Not an answer to your question, but interesting context for how eBooks work with libraries. And how not checking out a second time effectively is a loss of one “use” that may have directed the library to spend more on this book, rather than another. Sounds like you’re aware. Loved this Planet Money podcast
https://podcasts.apple.com/us/podcast/planet-money/id2907834...
lxgr
That link isn‘t working for me. Are you referring to this episode?
https://www.npr.org/2022/11/09/1135639385/libraries-publishe...
DanBC
> Is that a crime?
You didn't say where you are. Copyright infringement tends not to be a criminal offence, unless you're doing it as part of a business (so if you print out copies of that ebook and sell them) or you do so much of it you distort the market (you distribute so many copies of that ebook you destroy the profits that publisher might have made).
The rights holder can sue for their losses, but this tends to be the cost of the item, so they tend not to do that.
Tams80
That's really down to your personal ethics and how much you are willing to tolerate.
As you're not distributing it, the chance of someone coming after you for downloading a non-legal copy is almost non-existent, and pretty much 0 if you've taken some basic precautions.
It sounds like you've answered your own question there; that as you'd read the majority of the book, that there's no real loss to you finishing it for free.
LarryMullins
Of the past twenty ebooks I've downloaded, 18 have dead authors and only one is alive.
Unless you focus on the very latest releases, most authors are already dead. If you round up just about any list of great works from the 20th century, you'll find that most of those authors are dead and have been for years. If you like 19th century books, all of those authors are dead.
dlsso
Curious about the 20th guy who is neither dead nor alive.
baptiste313
Indeed the site seems to be down but you can use this site as an alternative:
Or else the project is opensource you can install it by yourself:
undefined
nyolfen
this is the result of the annas-archive people mirroring the contents of their crawling project onto ipfs -- huge props
marcol1n0
I got an error when visiting zlib.zu1k.com/.
Error code: 1020
Ray ID: 781a36f1b9b0babd
Country: IT
I got an error when visiting zlib.zu1k.com:
Error code: 1020 Ray ID: 781a37c3aeacbae1 Country: IT Data center: mxp03 IP: 31.157.83.62 Timestamp: 2022-12-30 10:51:01 UTC
Jamie9912
Access denied Error code 1020 You do not have access to zlib.zu1k.com.
The site owner may have set restrictions that prevent you from accessing the site.
Shadowgamer195
It is difficult to determine the quality of content on websites like z-lib and libgen because the only available indicators are size and a general description of the upload, which can make all content appear similar and leave users unable to differentiate between high and low quality content.
Thorentis
What exactly is Z-Library right now? The domains were seized, and there doesn't seem to be a way to access the library or add new books to it. Is it now simply a collection that exists alongside libgen? Is it being run elsewhere? Does the Ipfs mirror get updated from somewhere?
Zuiii
I've always been able to access it via tor which is operating normally.
Just be sure to confirm a disposable email address with them or they'll never let you in. Registering an account by itself is no longer enough when using tor and their website will block you and not tell you why.
netfortius
There is a discord server now, where you will find all your answers, plus some.
bscphil
If that's the case, then providing the answers should be pretty easy. Can you tell us what they are?
netfortius
Sorry - missed this. "The" answers to almost all the questions in this thread could not be consolidated in a single post. Looking in the dedicated channels on the discord server dedicated to this topic will allow searching for all related info. Unfortunately posting invites in a large community like HN could only endanger the well being of the discord site - wish there was a DM option in HN (that I know of ... maybe there is?!?)
Thorentis
Where does one find an invite to it?
netfortius
Unfortunately posting invites in a large community like HN could only endanger the well being of the discord site - wish there was a DM option in HN (that I know of ... maybe there is?!?)
Get the top HN stories in your inbox every day.
I like reading about IPFS, but I do not really have the time to learn about it and get involved.
Last I remember, Z-Library was having an issue scaling the DHS to handle the number of files [1]. Did those issues get resolved? How is it going now? Also, is there anything being done to ensure every file has seeders?
[1] https://news.ycombinator.com/item?id=33716560