Skip to content(if available)orjump to list(if available)

BitTorrent Site Magnet Database Archives Available via Torrent and Direct DL

gaius_baltar

Let's not forget one of the best approaches for indexing magnet hashes: https://btdig.com . It basically connects to the DHT and listens to queries. When a new one is found, it downloads the torrent data and indexes it.

r3trohack3r

DHT work is fun - you can scrape a lot of interesting content. BitTorrent and Tor both use DHTs for content discovery.

If you’re interested in building a minimum viable scrapper, this is a good reference: https://github.com/retrohacker/Taboo/blob/main/lib/index.js

The trick to content discovery on the DHT is to parrot any incoming request you receive. So if someone asks for a hash or announces a hash, you turn around and ask for that hash too. You can spin up multiple DHT nodes to increase your coverage of the table.

Some cool user experiences can be built off of this. I’ve discovered a tonne of interesting content using “BitTorrent Roulette” applications that download random hashes from the DHT - often times finding fascinating content from languages I don’t speak.

KennyBlanken

"BitTorrent Roulette" sounds like a wonderful way to end up with child porn on your computer.

CyanBird

Are people still so naive as to use torrent for that?

Well I'd guess I might be the naive, given the ample amount of honey pots and such

LeoPanthera

Wow, thank you for that. That's very clever and very useful.

It's a shame they use recaptcha, it's means it's impossible to script searches with something like Jackett.

e12e

well, you could run your own instance: https://github.com/btdigg-org/dhtcrawler2

tn1

Am I missing something or is the source not actually available in that repo? I only see the compiled .beam files

LeoPanthera

Ah, that is perfect. Thank you.

vinibrito

Well, one can write a bot that defeats recaptcha. Trivial, once you know what to do.

LeoPanthera

If "one" can write a bot that defeats recaptcha, what is the point of recaptcha? I think extraordinary claims require extraordinary evidence.

metadat

Badass. Thank you sir, madame, or they.

marci

> sir, madame, or they

Neither, I think it's a cylon.

metadat

I am assembling the grand mac daddy of all magnet link dbs / seeding systems. If you want to subscribe to updates or are otherwise interested please reply here so I can gauge interest. Will post to HN when ready. This is bigger than piracy, it's about free flow of information over time for humanity.

I am up for collaborating and sharing.

squarefoot

> https://btdig.com

First hint suggested: "interracial gangbang"

They might want to put those hints behind a NSFW warning; just my .2c.

killingtime74

Anyone who has spent more than 5 minutes in torrent-land has seen NSFW

jokoon

that link doesn't work

throwamon

I love what the Internet Archive does and I assume it pisses off many big fish. How do they get away with so much content? What guarantees are there that they're going to be able to preserve digital history in the long term?

baobob

Here is one of thepiratebay up to 2019: https://archive.org/details/ThePirateBayMarch2019

RachelF

Thanks, the one in the link is from 2016

banana_giraffe

A different torrent metadata archive (.torrent files) that's somewhat more up to date:

https://archive.org/details/torrent_metadata_archive_sample

(Though, in practice not as easy to download, you need to download a tar ball for each month, the naming is predictable, so you can at least script the download)

kingcharles

This is one of the biggest DHT search engines I know of.. though it'll just tell you if something exists or not, you have to find the magnet data somewhere else:

https://torrenthistory.org/

ReactiveJelly

Any way to know which ones might be under surveillance?

Right now I have to choose between darkweb piracy (Slow downloads, hard to find things, search services are slow) and clearnet piracy (Lots of popular shows are monitored by copyright bots), so I've been pirating less and less

flatiron

Pay $40 a year for a vpn. Use a docker container that has iptables set up to only use the vpn and put a torrent client. $40 a year and nobody will bother you. The person that a vpn provider will expose to lose their company base will be a terrorist not a torrentor

kblev

Or use a seedbox

Frotag

or set up openvpn on a dirt cheap vps based in romania or some shit

sergiotapia

I have fiber so a VPN just nukes my download speeds enough that I hated using it for torrenting.

Is this normal in your experience? I am/was a VPN noob.

oefrha

Do not route everything through the VPN then, only bind your BitTorrent client to the VPN’s network interface. Also has the advantage that if the VPN connection accidentally drops, your BT client doesn’t start transmitting on your residential and expose your IP.

unknownaccount

Get your own VPS and install OpenVPN server on it. Much faster than shared VPNs. I get gig speed on a $12/year VPS.

null

[deleted]

Nuzzerino

> Any way to know which ones might be under surveillance?

Yes, the answer is that all of them "might" be under surveillance. There's no way to guarantee that any of them aren't.

LeoPanthera

Content piracy is one of the very few valid uses of a public VPN provider service.

geranim0

I receive complaints all the time by my ISP for torrenting... they do nothing except frightening older people

schroeding

Not universally true though, and in some countries (like e.g. Germany) it's a really bad idea to torrent without a seedbox or vpn in another country.

ssl232

What's different about Germany when it comes to piracy? And what about using a PeerGuardian blacklist to block known trackers?

autoexec

If you're in the US I advise caution as ISPs are going to be increasingly aggressive about shutting off accounts. The media industry has demanded that repeated, unproven, accusations alone should be enough to force your ISP to close your account and never allowing you to open a new one. ISPs don't want to do that, but courts so far have been agreeing with the media industry on that. ISPs who don't do it, risk literal billions in fines.

IF you don't have multiple ISP options each offering reliable high speed connections in your area, you should be especially careful or you could find yourself with very limited access to the internet or even no internet access at all.

hombre_fatal

13 years ago, last I used them, Comcast would send an email with a list of the titles of the torrents it thought you downloaded.

Good enough to scare ~everyone by itself, though iirc they did threaten to halt your service, something I never decided to test out.

gabereiser

I tested it, it's legit, they will terminate your service.

doublepg23

Private torrent trackers. Check out r/trackers.

It requires time and effort though.

brnt

How does that help? YourISP is not looking at your HTTPS traffic, but your torrent traffic, right?

colinmhayes

I don't think it's the ISP monitoring for torrent traffic, but copyright holders honeypotting torrents and keeping track of the IP of seeders, maybe leechers too I guess. If the IP holder isn't on the tracker they don't see that you're pirating their stuff.

willhinsa

The ISP in general isn’t going looking for your torrent traffic. The DMCA groups are, and when they find it, they contact your ISP to enforce it. If it’s a private tracker, there’s a decent chance the DMCA groups’ tracking bots aren’t able to see that traffic.

doublepg23

Yes as the above commenters say the copyright holders (their enforcers really) don't have access to the private trackers swarm and will be unable to report your IP address.

password4321

You can hire a 3rd party to do the torrenting and then stream from them (put.io, etc.), but there's no guarantee they can't be compelled to hand over their logs.

slowbdotro

2016 database, someone needs to update it. And how does one even get all the magnet links like this?

nulptr

One way is querying the dht

jokoon

Funny, most of those torrent got a __padding_file, which comes from bitcomet, a bad torrent client... I think?

Other thing, I realized that there are less and less torrent indexes those last 5 years, they all end up offline or full of ads.

banana_giraffe

It's part of BEP-47 [1], an extension meant to make it easier to handle single file downloads and multiple torrents serving the same file. BEP-47 aware clients, which is most of them these days, I think, hide these files from the user.

[1] https://www.bittorrent.org/beps/bep_0047.html

duskwuff

> Funny, most of those torrent got a __padding_file

Those are supposed to be hidden by clients. They represent "free space" between files, allowing clients to download individual small files without having to also download the files adjacent to them.

smegsicle

yeah bitcomet had a similar, but incompatible scheme

5evOX5hTZ9mYa9E

How is this different from https://torrents-csv.ml/?

ranger_danger

seems to have a whole lot more results, for one

SahAssar

The one posted here is a dump from 2016 and https://torrents-csv.ml/ has imported piratebay dumps from then (probably from the same source). The difference is that https://torrents-csv.ml/ does health checks on the torrents and will remove dead ones which means that a lot of the torrents in this dump will be removed since they have probably died either before 2016 or during the last 7 years.

timetraveller26

Is the torrent link in the database?