- "Dude, we lost, like, everything. ;_;"
January 12, 2008: Catastrophic hardware failure strikes the Wiki, resetting all articles back to mid-October. There was great lamenting and scrambling by editors to save articles from Google's and Yahoo's caches.
This is their story.
Not to be confused with the (obviously less important) Great Crashes of 1929, 1983, 1996, 2007, 2029 or 2064.
The discussion below began in Contributor Announcements but was soon moved to and continued here.
The following pages were created and indexed here in the midst of the discussion and later deleted once the healing process had completed:
- HTML-to-Wiki format converter
-
Images We Miss
- Images were not destroyed, but their URLs were forgotten. Here, tropers appended requests for anyone who might remember what/where particular images had been. (This is before the Images List tool was created.) -
Pages Needing Formatting
- Index of pages whose content had been recovered, but badly—copy-pasting straight from caches and so forth. The HTML-to-Wiki converter was a huge help. -
Scavenger Hunt
- Index of pages containing the titles for all YKTTWs launched during the missing period, as well as the caches of various other index pages. Titles were blue-shifted as their caches were restored or crossed out if no cache could be found. -
Verify An Index
- Never really took off.
Fast Eddie: The Great Crash of '08 update: We're back to life, of a sort, for tropes. No way to sugar-coat it: We lost just about everything from mid-December through 12JAN. Backups were corrupted. As of about 20 straight hours of being into trying to put things back together, I'm a getting a bit cross-eyed and am going to grab a few Z's. There may be hope for the lost material, after my brain comes back online.
Tangent 128: Ouch. Yeah, I'd want a rest too.What kind of corruption are we talking about? If it's some sort of "database is glitched but the raw data's there" thing, maybe some of us could volunteer to pick through the files and rebuild stuff by hand?
Or are we talking line-noise-bad?
Heart Burn Kid: Well, thanks for doing what you could... I think I'll spend some time tomorrow trawling through Google's cache and seeing if I can save some of my edits (I just did just that for the WCW page).
Tangent 128: Oh, and insert Rousing Speech here. We will overcome! The Data Vampires shall rue the day they made us enemies! And such.
Morgan Wick: Shouldn't the backups be stored in a way that they wouldn't be affected by anything that would cause us to need them? Doesn't that defeat the point of backups? (Sorry, I love to nag)
You say mid-December, but everything in the Forums post-October is gone...
Heart Burn Kid: I'm wondering the same thing myself... please don't tell me the backup was stored on the server...
It looks like there was plenty of data loss for December and November on the wiki side, too; look at the Recent Changes page.
Cassius335: Hell of a way to start a year. On reflection, only losing a month or so doesn't seem so bad. At least we didn't lose everything since the beginning.
Lale: It sounds that way, but after browsing around, I found it amazing how much we write around here in a month...
Fast Eddie: Certainly the best idea right now for restoring a given article is to grab it out of the Google cache. I'll keep jamming away at the backups to see what can be recovered, but it looks slow and if-fy at this point.
Tech-y stuff: The biggest operational error made here (by me) was to take the backups on a database level, rather than on a table-level. Some tables (the non-article tables, BBS, YKTTW, etc) live on one disk, the one that didn't fail, and the others (articles, history) live on the one that did. There was corruption in the history table. The whole-database backup would not-so-cleverly aggregate all the data into one corrupted store and move it off to a safe place. A frustrating thing is that the per-transaction storage (each edit/addition to an article) which could be used to roll the article as close to its latest version as possible is the thing that was corrupt. Really rotten luck.
Going forward, backups will be made on a table level, using two techniques (as SQL and as a binary copy — we were just binary, before). Also, all transactions will be replicated to the new Search box so that there will be more backups per table to rely on.
Kizor: Eddie! Good to have you here, those of us laboring with cache-pulling (see
Scavenger Hunt Launches Part One) have been wondering about what you'd have to say. This is important: Is there any reason not to go all out and pull as many pages from the cache as possible? Can it interfere with retrieval or restoration on your end?
Silent Hunter: Nasty thing to happen. I'd help, but I don't know a thing about cache-pulling.
Where are our back-ups stored anyway?
Kizor: Further, what word of the mid-October to mid-December changes? Does the devastation stretch across a month or a quarter of an year? Should we restore non-trope pages (such as Made Of Win, avatar and the airbending etcetera) to where they were or under temporary names?
Citizen: Yeah, I'm wondering when would be the time to start revising pages again. Should I be waiting for you to restore pages from backup first?
Fast Eddie: Articles are pretty much as restored as they are going to get, so revisions can go forward. YKTTW and the Forum have a better chnace of being brought up to date by stuff I'm working on now. New entries and edits there are safe, but reconstruction-type stuff should hang on for a bit.
How to Cache-Grab: Search Google for the page name. If you search as a Wiki Word, we're usually on the first page. Select the "cached" link on the item. That will take you to the text as it was a few days ago.
Attilargh: Is there any way to get one's hands on the markup? Restoring pages like the series will otherwise be rather time-consuming.
Silent Hunter: Don't think so. Ignore formatting at the moment, IMO. Save our entries! Copy and paste the lot.
Tangent 128: Restored as much of World Creation Project as I could. I'm pretty the missing pages were never filled in to begin with, but if you wrote something there, try to check.
Silent Hunter: Calling it a night now (it's 10.05pm in London). I'll be back tomorrow, but we might be done by then. Good work everyone!
Adam850: Someone should edit the robots.txt file here to stop Google from cache-ing, so we can preserve the good cached pages.Plus, Is there a way to look at a listing of the images that reside on the server, so we can find lost images?
Tangent 128: OK! Progress on the formatting job! I have a basic html-back-to-wiki thing going. Paste the html in, click the button, copy out. (Supports bold, italic, spoilers, and rules, but not colors, WMG, or categories. Also, links are translated as external links, not wiki words. Still, should be a big help.)
Citizen: It would work much better if you didn't translate internal links as external. Leave off the http://www.tvtropes.org/pmwiki/pmwiki.php/Main/ part.
Morgan Wick: That might be harder than you make it sound, just because of the extra coding needed to separate internal links from external, not to mention dealing with non-Main namespaces.
I just had a crazy idea: Would it be too insane to see if the Internet Archive could maybe lend us a hand? (Normally the Archive only allows access to archives more than six months old, but there's a part of me that thinks they just might be willing and able to lend us more recent files.)
How soon before YKTTW and the Forum are restored?
Mith: I now have a local copy of the cached versions of everything that was in the category Speculative Fiction Tropes and its subcategories (so none of that is time-sensitive) and am working on more. Unfortunately Google doesn't like retrieving cached pages too fast, so it goes slowly. Move this to
Scavenger Huntif you think it's more helpful there.
Fast Eddie: No joy on restoring YKTTW or Forum data. Actually, we even lost the structure of the database tables for YKTTW. I've recreated the YKTTW structures (going backward from the SQL queries). The Forum data ... if it hadn't been running on a another box before this one, we wouldn't have what we've got. As far as restoration goes, then, this amazing pulling together of people to plunder the Google cache will have to suffice.
No kidding, folks, this effort to restore things that just sprang up among the Tropers ... just amazing. There's hope for the Internet, yet.
I'll be closing down the webserver at midnight to install the new backup goodies. Should take about 15 minutes or so.// later: That's installed.
Silent Hunter: HTML from cache doesn't translate for index mark-ups.
Citizen: Go to the forum and use my HTML pages to download all the cache for the Scavenger Hunt. I can't do it alone, Google keeps banning me for spam downloading...
Medinoc: Sound-Effect Bleep! Google has "updated" His Dark Materials to the "lost" version, and I'm afraid many pages suffered that fate...
Daviot: Good news, tropers. Most of my edits were before the archive date >_<;. Regardless, I've got caches of them, and will be fixing them shortly. Midonin, I only have the 7 Jan. Google cache for the R+V page. How do you want to go about tweaking it?
billythehick: January 12th was my birthday. Not the best of omens... Still, my sister holds the record, her birthday heralded the start of the gulf war.happy birthday to me...
John Quixote: It's amazing to see such a small group of people working together to save a great site. This community is so much better than some others I visit. I just wish I had known about this earlier so I could have helped more; it looks like most everything has been saved, and everything that hasn't is lost forever (or for at least half a year). But seriously, just saying, thank you, everyone, for not only bringing this amazing site back, but for setting a good example for the rest of the 'net.
Demetrios: I spent three hours last night reviving the Nightmare Fuel pages. But it was worth it.
Donomni: Adding to the watchlist doesn't seem to be working right at all... took me a few dozen tries to re-add Final Fantasy XI to my list again.
Insanity Prelude: Brought back a few pages, but most of the work had been done already when I found out about it. Incidentally, the index markup wasn't always coming out right, even when I used the handy little dropdown list here instead of trusting to the HTML-to-Wiki converter.Amazing job. You guys have the patience of gods~
Looney Toons: Has anyone had any success recovering any lost images?
Adam850: Is it possible to get a listing of the images directory?
fleb: If you successfully find a Google or Yahoo! cache, you can view-source and search for the '<img', and check the 'src=' URL. All the images are still exactly where they were in antediluvian times. And @Medinoc: I've restored His Dark Materials from Yahoo!'s cache. It seems they're a bit slower than Google, which is a good thing now.
Tropers/Da_Nuke: I must say, I'm touched. After doing my part to restore this site to its old pre-crash glory, I found out that pretty much every single page has been restored. This is by far the best wiki in the universe. T_T ^_^
On a side note, as of right now I'm downloading an entire archive of the site with HT Track. Every single page will be saved to my hard drive. Building a full backup is kinda hard because that means leaving my computer on for like 3 full days, and my parents are really fussy with the power bill, but I'll try my best. It's the least thing I can do for this site.
Kizor: It's now January 17th, 3:21 AM by your puny standards of Pacific Standard Time, and for the first time since they were started I see no page relating to the repairs in the 100 most recent changes.
Pteryx: Whew, there — I've saved one index and its members, however small (This Trope Name References Itself). Goes to show the job's not quite done yet, though. Who else wants to
Verify An Index?
Donomni: Checked a bit: It seems the only articles I have problems putting on the watchlist are ones I had before the Crash on after the cutoff date. Tried clicking remove as well... no go. Any ideas?
Po8: Please see my offer of assistance with offsite backup in the Wiki Tech Wish List. I'm pretty confident I can set up something that will be pretty regular and pretty complete, with help from those "on the inside".
Heatherly: I'm going through to try and add any edits I'd contributed (I tend to have them betaread in IMs before pasting them here, and my IM program automatically logs any IMs I make). I wanted to help add entries, but looks like almost everyone had gotten that covered! I found that tropes I've written (which I had backed up on my own hard drive) were already back before I even knew they were missing. This was so incredibly awesome. I checked the page histories so I could credit the restorers on my contributor page.
Code Man 38: Something needs to be done about Mood Dissonance and Soundtrack Dissonance, but alas, I'm too much of a Genius Ditz to take the task on myself without screwing it up somehow. The former, if I recall, was renamed to the latter so that the former could become more generalized, but we currently have two divergent versions of the same article...
Silent Hunter: When can we start getting stuff from archive.org?
Ansem Paul: Oh man i feel so silly... I thought someone had removed my edits from some stupid reason agisant me, so I yelled at them. Opps.I guess the universe doesn't revolve around me...
