Get the top HN stories in your inbox every day.
gaius_baltar
r3trohack3r
DHT work is fun - you can scrape a lot of interesting content. BitTorrent and Tor both use DHTs for content discovery.
If you’re interested in building a minimum viable scrapper, this is a good reference: https://github.com/retrohacker/Taboo/blob/main/lib/index.js
The trick to content discovery on the DHT is to parrot any incoming request you receive. So if someone asks for a hash or announces a hash, you turn around and ask for that hash too. You can spin up multiple DHT nodes to increase your coverage of the table.
Some cool user experiences can be built off of this. I’ve discovered a tonne of interesting content using “BitTorrent Roulette” applications that download random hashes from the DHT - often times finding fascinating content from languages I don’t speak.
KennyBlanken
"BitTorrent Roulette" sounds like a wonderful way to end up with child porn on your computer.
CyanBird
Are people still so naive as to use torrent for that?
Well I'd guess I might be the naive, given the ample amount of honey pots and such
LeoPanthera
Wow, thank you for that. That's very clever and very useful.
It's a shame they use recaptcha, it's means it's impossible to script searches with something like Jackett.
e12e
well, you could run your own instance: https://github.com/btdigg-org/dhtcrawler2
tn1
Am I missing something or is the source not actually available in that repo? I only see the compiled .beam files
LeoPanthera
Ah, that is perfect. Thank you.
ghostkba
[flagged]
vinibrito
Well, one can write a bot that defeats recaptcha. Trivial, once you know what to do.
LeoPanthera
If "one" can write a bot that defeats recaptcha, what is the point of recaptcha? I think extraordinary claims require extraordinary evidence.
metadat
Badass. Thank you sir, madame, or they.
ghostkba
[flagged]
marci
> sir, madame, or they
Neither, I think it's a cylon.
metadat
I am assembling the grand mac daddy of all magnet link dbs / seeding systems. If you want to subscribe to updates or are otherwise interested please reply here so I can gauge interest. Will post to HN when ready. This is bigger than piracy, it's about free flow of information over time for humanity.
I am up for collaborating and sharing.
squarefoot
First hint suggested: "interracial gangbang"
They might want to put those hints behind a NSFW warning; just my .2c.
killingtime74
Anyone who has spent more than 5 minutes in torrent-land has seen NSFW
jokoon
that link doesn't work
throwamon
I love what the Internet Archive does and I assume it pisses off many big fish. How do they get away with so much content? What guarantees are there that they're going to be able to preserve digital history in the long term?
baobob
Here is one of thepiratebay up to 2019: https://archive.org/details/ThePirateBayMarch2019
RachelF
Thanks, the one in the link is from 2016
banana_giraffe
A different torrent metadata archive (.torrent files) that's somewhat more up to date:
https://archive.org/details/torrent_metadata_archive_sample
(Though, in practice not as easy to download, you need to download a tar ball for each month, the naming is predictable, so you can at least script the download)
kingcharles
This is one of the biggest DHT search engines I know of.. though it'll just tell you if something exists or not, you have to find the magnet data somewhere else:
ReactiveJelly
Any way to know which ones might be under surveillance?
Right now I have to choose between darkweb piracy (Slow downloads, hard to find things, search services are slow) and clearnet piracy (Lots of popular shows are monitored by copyright bots), so I've been pirating less and less
flatiron
Pay $40 a year for a vpn. Use a docker container that has iptables set up to only use the vpn and put a torrent client. $40 a year and nobody will bother you. The person that a vpn provider will expose to lose their company base will be a terrorist not a torrentor
sergiotapia
I have fiber so a VPN just nukes my download speeds enough that I hated using it for torrenting.
Is this normal in your experience? I am/was a VPN noob.
oefrha
Do not route everything through the VPN then, only bind your BitTorrent client to the VPN’s network interface. Also has the advantage that if the VPN connection accidentally drops, your BT client doesn’t start transmitting on your residential and expose your IP.
unknownaccount
Get your own VPS and install OpenVPN server on it. Much faster than shared VPNs. I get gig speed on a $12/year VPS.
undefined
celdon25
> Any way to know which ones might be under surveillance?
Yes, the answer is that all of them "might" be under surveillance. There's no way to guarantee that any of them aren't.
LeoPanthera
Content piracy is one of the very few valid uses of a public VPN provider service.
geranim0
I receive complaints all the time by my ISP for torrenting... they do nothing except frightening older people
schroeding
Not universally true though, and in some countries (like e.g. Germany) it's a really bad idea to torrent without a seedbox or vpn in another country.
ssl232
What's different about Germany when it comes to piracy? And what about using a PeerGuardian blacklist to block known trackers?
autoexec
If you're in the US I advise caution as ISPs are going to be increasingly aggressive about shutting off accounts. The media industry has demanded that repeated, unproven, accusations alone should be enough to force your ISP to close your account and never allowing you to open a new one. ISPs don't want to do that, but courts so far have been agreeing with the media industry on that. ISPs who don't do it, risk literal billions in fines.
IF you don't have multiple ISP options each offering reliable high speed connections in your area, you should be especially careful or you could find yourself with very limited access to the internet or even no internet access at all.
hombre_fatal
13 years ago, last I used them, Comcast would send an email with a list of the titles of the torrents it thought you downloaded.
Good enough to scare ~everyone by itself, though iirc they did threaten to halt your service, something I never decided to test out.
gabereiser
I tested it, it's legit, they will terminate your service.
doublepg23
Private torrent trackers. Check out r/trackers.
It requires time and effort though.
brnt
How does that help? YourISP is not looking at your HTTPS traffic, but your torrent traffic, right?
colinmhayes
I don't think it's the ISP monitoring for torrent traffic, but copyright holders honeypotting torrents and keeping track of the IP of seeders, maybe leechers too I guess. If the IP holder isn't on the tracker they don't see that you're pirating their stuff.
willhinsa
The ISP in general isn’t going looking for your torrent traffic. The DMCA groups are, and when they find it, they contact your ISP to enforce it. If it’s a private tracker, there’s a decent chance the DMCA groups’ tracking bots aren’t able to see that traffic.
doublepg23
Yes as the above commenters say the copyright holders (their enforcers really) don't have access to the private trackers swarm and will be unable to report your IP address.
password4321
You can hire a 3rd party to do the torrenting and then stream from them (put.io, etc.), but there's no guarantee they can't be compelled to hand over their logs.
ghostkba
[flagged]
slowbdotro
2016 database, someone needs to update it. And how does one even get all the magnet links like this?
nulptr
One way is querying the dht
jokoon
Funny, most of those torrent got a __padding_file, which comes from bitcomet, a bad torrent client... I think?
Other thing, I realized that there are less and less torrent indexes those last 5 years, they all end up offline or full of ads.
banana_giraffe
It's part of BEP-47 [1], an extension meant to make it easier to handle single file downloads and multiple torrents serving the same file. BEP-47 aware clients, which is most of them these days, I think, hide these files from the user.
duskwuff
> Funny, most of those torrent got a __padding_file
Those are supposed to be hidden by clients. They represent "free space" between files, allowing clients to download individual small files without having to also download the files adjacent to them.
smegsicle
yeah bitcomet had a similar, but incompatible scheme
5evOX5hTZ9mYa9E
How is this different from https://torrents-csv.ml/?
ranger_danger
seems to have a whole lot more results, for one
SahAssar
The one posted here is a dump from 2016 and https://torrents-csv.ml/ has imported piratebay dumps from then (probably from the same source). The difference is that https://torrents-csv.ml/ does health checks on the torrents and will remove dead ones which means that a lot of the torrents in this dump will be removed since they have probably died either before 2016 or during the last 7 years.
timetraveller26
Is the torrent link in the database?
ghostkba
[flagged]
Get the top HN stories in your inbox every day.
Let's not forget one of the best approaches for indexing magnet hashes: https://btdig.com . It basically connects to the DHT and listens to queries. When a new one is found, it downloads the torrent data and indexes it.