Get the top HN stories in your inbox every day.
tkgally
raybb
Donations are fantastic but if you have engineering (or project management, design, etc) skills spending just 1 hour a week contributing to their open source goes a very long way!
Open Library in particular has a very active repo with lots of volunteers, a weekly community call, and a rather accessible codebase. https://github.com/internetarchive/openlibrary
If anyone knows webpack well would LOVE to have this dev-facing issue resolve to auto reload CSS https://github.com/internetarchive/openlibrary/issues/4955
dailyanchovy
Alright, I'm new to pull requests but I had a go at it! https://github.com/internetarchive/openlibrary/pull/5451
state_less
I sent my bones or clams or whatever you call them.
Will send moore when I have more and when I've learned to be more generous. It's good to know that you're near Internet Archive.
But, oh, what a wonderful feeling
Just to know that you are near
Sets my a heart a-reeling
From my toes up to my ears
-Bob Dylan, The man in me
mkaufman
state_less sounds like a Little Lebowski Urban Achiever.
state_less
Little Lebowski Urban Achievers - inner city children of promise but without the necessary means for a - necessary means for a higher education. So Mr Lebowski is committed to sending all of them to college.
walterbell
Anyone know the relative budgets/donations/staff of Wikipedia vs. Archive.org?
tkgally
ProPublica has a database of tax filing information for nonprofits.
Internet Archive: https://projects.propublica.org/nonprofits/organizations/943...
Wikimedia Foundation: https://projects.propublica.org/nonprofits/organizations/200...
Mozilla Foundation: https://projects.propublica.org/nonprofits/organizations/200...
Electronic Frontier Foundation: https://projects.propublica.org/nonprofits/organizations/430...
jdc
Wow, never thought I'd see Wikimedia at more than 4x the budget of Mozilla!
joe_the_user
Nice they're there. At the same time, it's amazingly easy for content to be removed from there - if someone objects or even if things are murky.
For example, all content from the old ezboard site was been removed based on the configuration of the current URL owners' robots.txt, and current URL owner is just a domain parker. Ezboard hosted a lot of content back in the day.
1vuio0pswjnm7
This is an old problem I could have sworn there were promises they were going to change their procedures.
The question I have is how fast is the content removed after the domain name registration changes, i.e., is there is a window of time between the appearance of a new robots.txt and the next scheduled crawl, and if so, is it be possible to "rescue" the content, as ArchiveTeam would do, during that window, before it disappears.
If this is possible, there could be a service for monitoring changes to domain name registrations for sites that have large amounts of historical content. I would happily volunteer to set up such a service.
joe_the_user
Well, "complain on hn" has been a way to get stuff from Google. Maybe someone at archive will notice this thread.
hidden-spyder
I'm curious. What changes has Google made due to complains on HN?
SilverRed
Hopefully it is just hidden and not deleted. But this is the main reason why alternative archive sites exists which ignore the original posters requests. Frequently used to archive posts from public figures which are suspected to be attempted to be scrubbed later.
techrat
> Hopefully it is just hidden and not deleted.
Hidden. Even when you request for them to remove stuff.
Had domain, stuff got archived, asked for them to remove it, added robots.txt. Domain lapsed. Someone else picked it up. their robots.txt now permissive, old stuff that I requested for them to remove is now visible.
throwslackforce
That's insidious. Does it even make sense to revive data that has been removed based on current configuration?
Even if the owner is the same, allowing the site to be archived going forward isn't the same thing as permitting it retroactively.
mavhc
Wonder how you'd overcome that flaw, is there a history of domain name ownership?
fwn
As far as I was able to experience it's just hidden and not deleted.
I have to keep an old domain indefinitely to host a robots.txt just to keep sensitive personal data hidden that little me foolishly published on the open internet.
But I'm not complaining. The internet archive is a great gift. Using it with a bookmarklet really feels like a super power.
mercora
it sounds a lot like this would need some kind of delegation mechanism where you could point to a different URL in-time before abandoning the place. or maybe some kind of sealing using a cryptographic function that lets you proof your are the owner of the current/previous content and also would proof you are not the owner of the newer content while this proof could be used to release the ban if ever needed.
joe_the_user
Got any examples of these alternative archives?
SilverRed
This is the main one https://archive.is/
From the FAQ, they do not respect robots.txt since they only archive on request by a user and they do not remove archives unless they contain illegal content.
mellosouls
A couple of pointers to the wider world of web archiving:
https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...
v0x
There’s archive.is but I get the sense that the major use case for that is getting around paywalls as opposed to permanently archiving a page - indeed, since they host content that the site owner probably doesn’t want them to, it would stand to reason the service would not be likely to stand the yet of time. But I could be wrong.
Jiro
They actually posted about this in 2017: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... . At the time it sounded like they might change their robots.txt policy. I guess they never followed up on it.
(I checked and ezboard is still excluded.)
DonHopkins
A search of youtube for "wayback machine" produces pages of stuff about the Internet Archive, and only the 24th result has anything to do with the origin of the term.
People who didn't spend their Saturday mornings glued in front of the TV screen as a child of the 1970's might not remember how American kids learned about history back then:
Peabody's Improbable History - Surrender of Cornwallis
Peabody and Sherman travel back to October 19, 1781 to witness when Cornwallis surrendered for Washington. However, when they got there, then he didn't show up.
https://www.youtube.com/watch?v=3E8zmaOiCVw&ab_channel=bullw...
ConceptJunkie
Actually, "The Bullwinkle Show" premiered in 1959, but I discovered it in the early 70s, as I suspect you did.
pwdisswordfish0
FYI, it's not nowadays obscure. There's a current series The Mr. Peabody & Sherman Show on Netflix, and Hollywood made a Mr. Peabody and Sherman movie (by Dreamworks) in 2014.
elric
Without trying to be contrarian, I don't think that everything should be archived. Random tweets, random blog posts, random personal web sites. Let them wither and die and be forgotten. Notable content by notable people? Sure.
Everyone else ought to have the right to be forgotten, including some drunk tweet they wrote 10 years ago and regret, or an old personal page which contained too much PII.
Archive no longer has a way to opt-out, which is bad enough, but I still think they should be opt-in.
jl6
Perhaps the Internet Archive could do more to help people who find their personal/sensitive/embarrassing content made available in perpetuity (I’m not sure exactly what they could do), but it’s incredibly valuable to have archiving on by default. The voice of un-notable people is underrepresented in every field of study, and the voice of notable people tends to get preserved in other ways anyway.
jjkaczor
It is helpful to get the perspectives of "un-notable" people from a historical perspective.
For example - the graffiti at Pompeii is interesting (and is pretty much at the same "quality bar" as Twitter):
https://www.theatlantic.com/technology/archive/2016/03/adrie...
https://kashgar.com.au/blogs/history/the-bawdy-graffiti-of-p...
jacquesm
You never know in advance what will and what will not be worthwhile archiving, you only know that at some unspecified point in the future.
elric
That's lovely from the perspective of a historian 200 years in the future. But it does not nothing to alleviate the pain of people living in the present. People whose prospective employers comb over their embarrassing past. Or bullies. Or any other number of evildoers whose life is made easier by unfettered access to indelible information.
pwdisswordfish8
I don't like your proposed solution that uses right to be forgotten as a blunt instrument to paper over serious problems, to used a mixed metaphor. When you appeal to the right to be forgotten to fix problems like people and "prospective employers comb over their embarrassing past", it's a way of throwing up your hands and neglecting deeper issues.
In a right-to-be-forgotten world, the way it would end up going is:
1. problematic potentates punish pitiable proles
2. someone invokes right to be forgotten
3. this is considered "good enough"
4. the problem conditions that allowed #1 to fester remain uncorrected
I feel this way about a lot of stuff these days (especially the where the erosion of the tenets of a liberal society is involved), where people argue vociferously for a "solution" that can at best be considered an indirect way of handling the problem. You see this with a lot of contemporary calls for the dismantling of tradition of free speech/free inquire/freedom of association, for example. People end up chafing in the direction of proposals that have dual-use effects in the first instance and perniciously "null" effects in the second instance.
stared
Most of history we know is from the perspective of the wealthies 0.1%. Even though the Internet is still biased towards the wealthier and more educated, having a history of the wealthies 10% would be enormous progress.
X6S1x6Okd1st
Historians spend a lot of time pouring over minutia from non-notable people. There is plenty about the world that doesn't seem worth writing down, but can be intuited from tangential texts.
The costs seems low enough to just keep it.
dogorman
From what I understand, present day historians are sitting on a huge pile of cuneiform tablets that have yet to be transcribed or translated because there is much more material than there is interest/manpower.
Of course, to your point, they do keep it around. They don't just throw it in the trash.
jacobolus
Only a tiny number of people in the world can read Sumerian or Akkadian, even for them the process is slow and error prone because we are missing a lot of the original context, and they have better things to do than skim through piles of delivery receipts.
* * *
It would be pretty neat if someone could figure out how to OCR all the cuneiform tablets and turn them into something searchable.
ignoramous
> Random tweets, random blog posts, random personal web sites. Let them wither and die and be forgotten. Notable content by notable people? Sure.
Well, it isn't named the Internet Encyclopedia, for a reason.
> Without trying to be contrarian, I don't think that everything should be archived
It isn't contrarian. The deletionists are seemingly the majority. It is contrarian to in fact archive all. the. things.
generationP
Who decides what "notable" is? I frequently use the Archive to find old academic grey lit (preprints, lecture notes, newsgroup posts, etc.). Much of it is on "random" blog posts and personal websites. Even the authors aren't usually notable by Wikipedia standards. Yeah, there is some PII on those pages, but also treasures of useful information.
spiritplumber
The Internet Archive location is beautiful. it's a church that has been partially turned into a server farm. Big Neuromancer energy when you go inside and look.
DoingIsLearning
I was too curious not to look this up:
https://www.atlasobscura.com/places/internet-archive-headqua...
N3cr0ph4g1st
Did a hackathon there in 2016, it is so cool!
ignoramous
A little known trivia: Apache Hadoop (and the multi-billion dollar open source big-data ecosystem it spawned) was worked upon at first at Internet Archive [0].
Speaking of billions: According to Kahle, Alexa Internet's compute infrastructure informed Amazon's take on IaaS (AWS) [1].
Another perhaps lost nugget is Amazon once funded (either in part or in full) the development of the Wayback Machine, Internet Archive's most impactful product. In addition, till date (if I'm not mistaken) Amazon continues to donate data it fetches from Alexa Toolbar installations to the Wayback Machine.
dleslie
I love the Internet Archive; I worry that its utility will wane as content becomes more dynamic than static. What does it mean to archive the experience of scrolling through a social feed?
petertodd
The paid, legal-oriented, archiving service Perma.cc that Harvard Law runs actually lets you upload your own PDFs and screenshots in addition to allowing Perma.cc's bots capture webpages. Of course, since you could upload anything the difference is made clear in the UI.
In a legal context, simply attesting to the validity of a screenshot is really common. So when that functionality is used Perma.cc is operating more as a permanent file storage service than a trusted archive.
Regardless, this does go a long way to solving the problem of dynamic sites.
cxr
> actually lets you upload your own PDFs and screenshots in addition to allowing Perma.cc's bots capture webpages
FWIW: the Wayback Machine is just one part of the Internet Archive. The quoted bit accurately describes things you can do with an archive.org account, too. Readers here may be familiar with the archive.org-affiliated effort by a team specifically working to recreate the playability of old PC (and otherwise) video games with JSMESS.
> this does go a long way to solving the problem of dynamic sites
Maybe, but the "dynamic" aspect that I'm sure the other person had in mind doesn't have much to do with the D in DHTML so much as it has to do with the dynamism that arises when you have a smart server responding to requests from a fat, JS-powered frontend. It would be possible to accurately model this in and execute it from a series of static assets, in some cases, but it's rarely done.
Even many sites built with static site generators today are not going to be usable in the future. There's too much tight coupling to the environment/deployment configuration and not enough semantic richness to properly hint to the crawler what resources are necessary to archive. In the heydey of XML, it used to be a big deal to strive for machine readable documents. Today's resume-driven development-obsessed webdevs effectively cast a vote of no confidence even in HTML, doing an end-run around it daily, and figuratively holding up a middle finger to the Principle of Least Power.
https://www.w3.org/DesignIssues/Principles.html#PLP
To some extent, even a bunch of the projects associated with TBL's Solid initiative are guilty of doing the same.
musicale
> This library would have all the published works of humankind. This library would be available not only to those who could pay the $1 per minute that LexusNexus charged, or only at the most elite universities. This would be a library available to anybody, anywhere in the world. Could we take the role of a library a step further, so that everyone’s writings could be included–not only those with a New York book contract? Could we build a multimedia archive that contains not only writings, but also songs, recipes, games, and videos?
For every Sci-Hub trying to create the library of Alexandria, there's an Elsevier trying to burn it down.
Current copyright law is largely on the side of the arsonists rather than the archivists.
(note: recipes are not copyrighted, though cookbooks are)
akkartik
I wish IA hosted Usenet archives. Even if they stopped at year 2000 and didn't update further.
cxr
Anything particular about Usenet that makes it attractive? Modern mailing lists (and even ones that are now dead but accessible) are a treasure trove of information, and I've often privately mulled over my concern that there doesn't seem to be a concerted effort to save them. Having tried to track down old copies of the Linux mailing lists, and having run into indicators of others' quests (and dead ends), I know how fragile the situation is.
As I've recently come to understand, the Internet Archive itself used to have its own mailing lists for handling discussion, which interestingly enough seem to no longer be accessible—perhaps even lost.
generationP
They host some: https://archive.org/details/usenet
As far as my own old posts are concerned, it looks complete :) But it isn't easily findable or searchable; the intended way of interaction is apparently to download an entire hierarchy and grep.
akkartik
Oh good to know! That's adequate.
ZeroGravitas
This is a bit of a tangent, but there was mention of non-advertising based funding models.
Is anyone working on an advertising model that achieves the basic goal of advertising, but without the centralising aspect which seems to be the root cause of many of the issues? Giant monopolies are always going to subvert regulation but the same industry as disconnected units might be easier to police. Obviously you can just try to split them up or limit their size with regulation after the fact but a good technical basis might help out.
Does web advertising just not make sense unless you can amass lots of private user data and track people across the web? If so can we subcontract that data to smaller companies we can trust with our data and effectively punish if they break the rules?
coldpie
> Does web advertising just not make sense unless you can amass lots of private user data and track people across the web?
It does make sense, but it has to compete with the invasive-style advertising, and it will always lose. If you want the "good advertising" you have to kill the "bad advertising".
soheil
For anyone interested I made a quick and dirty way to pull up the archive of any URL by prefixing it with arxiv.link
e.g.,
fouc
Nice, thanks! Ever since I started using archive.is last year, I've always wanted something like this! Could be paired with a bookmarklet too.
javascript:(function(){window.open('http://arxiv.link/'+location.href)})();
alexislours
You have always been able to do this with archive.is, I’ve been using this bookmarklet for a quite some time.
javascript: (() => { window.open("https://archive.is/" + window.location.href, '_blank')})();
fouc
I meant I wanted to go directly to the most recent result in the wayback engine for the given page.
codethief
archive.is is not related to the Internet Archive's Wayback Machine, though, is it?
johtso
Is there an advantage compared to just putting `web.archive.org/web/` before the url?
fouc
Oh interesting, http://web.archive.org/web/https://news.ycombinator.com also works, didn't know that.
soheil
Didn't know that, thanks. I guess just shorter.
ipsum2
a little confusing since arxiv refers to a popular research paper archive. Useful project though!
causi
As a question related to archival, what's the best tool for local archiving? HTTRACK is getting long in the tooth and it just work for all the dynamic content on modern web pages.
Get the top HN stories in your inbox every day.
In his reflections, Brewster Kahle mentions his goal of creating “a library available to anybody, anywhere in the world.” He doesn’t mention, though, the costs of making that library available to the world for free or the fact that the Internet Archive accepts donations. So I will:
https://archive.org/donate/