Help save Gplus content to the Internet Archive! See within - it's a 400M download. You'll be running a virtual machine which grabs G+ pages and sends them to the Internet Archive to publish later. You need to leave it running of course, and every five mins it checks back for new work packages. You need 60G of free disk, which I don't have, but I ran it anyway.
(There's a command line version to run directly, if you prefer.)
Hat tip to Edward Morbius
https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior
(There's a command line version to run directly, if you prefer.)
Hat tip to Edward Morbius
https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior
What exactly would this be saving? What if I don't want any data, including my public posts, saved?
ReplyDeleteChris S that is why I am methodically deleting my content ahead of sunset. I don't feel a need to archive my content - but others used G+ more as I use my blog.
ReplyDeleteToo late, Chris S:
ReplyDelete"Saved 103 times between November 7, 2013 and May 25, 2017."
If you put something up on the public Web, it's quite possible that copies of it will be around forever.
Ed S Valid, but I only really gave the right to Google to archive it. I'm just wondering how all this will work out, since I'm not aware of any other social networks being archived.
ReplyDeleteThe archive team often tries to archive sites which have announced impending closure. And the Internet Archive archives zillions of public pages every day - including those 103 saves of your profile page. If it's public, the archivists want a copy.
ReplyDeleteI've a feeling the Internet Archive gets some sort of exemption as a library.
Chris S Multiple other sites have been. archived:
ReplyDeletearchiveteam.org - Archiveteam
Flickr, Geocities, Tumblr.
Even Facebook is on their watchlist:
https://www.archiveteam.org/index.php?title=Alive..._OR_ARE_THEY
https://www.archiveteam.org/index.php?title=Deathwatch
Personally (and acknowledging/echoing that our wants and needs vary), I would be thrilled if my intentionally public google posts were around forever. Great thing! However all my efforts have been in backing up all I can - intact (.json etc.) and a la cart (images, comments). Between my own backups & multiple sites I've posted to (including my own), it may be the case that long after I'm gone, Google+ alone is still showing some of my stuff. If so, IMM, a blessing! So I see no purpose in deleting what I put up intentionally, but respectful of differing contexts & perspectives. I am only planning on re-posting what's not up elsewhere now, with existing or newly archived components from G+.
ReplyDeleteSome save data. Some save images. Some save whole file systems. Some keep only memories. "YMMV". It's a wonderful thing IMM that even under duress, we have options.
Cool. Just looking to see if there was precedent for archiving social networks as well as possiblities for opt-out. I think they're reasonable questions.
ReplyDeleteChris S They are.
ReplyDeleteNB: This project is NOW RUNNING. I'm working on an announcement post.
ReplyDeleteThe tracker (accessible from the ArchiveTeam page above) shows progress being made.
John Lewis Filip H.F. Slagter Julian Bond ping.
Edward Morbius Just emerged from the rabbit hole again without an answer. Where's the G+Archiver tracker?
ReplyDeleteHmm, yes, I should join that. Though since I'll prob also be scraping content directly, I might need to run this from a secondary IP.
ReplyDeletetracker and leaderboard:
ReplyDeletetracker.archiveteam.org - Googleplus tracker Dashboard
8766GByte saved so far
Julian Bond Answers to all that and more:
ReplyDeleteold.reddit.com - Saving of public Google+ content at the Internet Archive's Wayback Machine by the Archive Team has begun
Note that the tracker itself only shows 1/5oth the total project at any one time, for reasons explained in the Reddit post.
Chris S If you are very concerned about the archive, delete your account. This is your only real possibility to avoid the archiving from this project. That doesn't mean you will be successful. You could already be archived or Google may one day allow people to search the archive of old users. If you are in the EU, you can petition for the "right to be forgotten." And that is a long legal battle, but if you are adamant about it then this is an option.
ReplyDeleteJohn Lewis I'm asking the questions to ask the questions. I'm not that concerned about my data. Although I do have another question pertaining to data. What about posts that are not public but shared to those archiving them? Will those be picked up as well? I imagine posts not shared with those archiving them simply cannot be picked up and will be "lost forever."
ReplyDeleteJohn Lewis Pertaining to the right to be forgotten, I work at a rather large company with international components. We're setting up one of our cloud based systems to be compliant with that, and we suspect that we'll have to set that up for the US in the next few years because we expect it will be picked up here as well. Curious to see how that effects things when/if it happens.
ReplyDeleteChris S since the Warrior runs in a separate virtual machine, it should not be affected by which user runs it, as (AFAIK) it doesn't offer the user to log in to their Google account. Therefore it should only be picking up public content.
ReplyDeleteFilip H.F. Slagter Thank you for the information. Great to learn how these things work.
ReplyDeleteEd S Ed, looks like you quit last night. We need every bit of help we can get. Long way to go. I'm Shuttersparks on the leaderboard.
ReplyDeleteRestarted the machine...
ReplyDeleteI wish I had any kind of bandwidth....
ReplyDeleteEd S Hey, hey, I see you're moving again. Yay. Thank you. Yeah, thus far, we've only captured 81 terabytes out of an estimated 1800 TB.
ReplyDeleteJohn Lewis Use what you have. Every work unit counts.
ReplyDeleteEd S Your machine seems to have frozen up some time ago. ;-)
ReplyDeleteWe're now at 89 TB out of 1800 TB. Looooong way to go.
C'mon people. We can do this but we need 6 to 10 times the amount of participation we have now in order to get this done by April 2.
Install the software and leave your machine running 24/7. Please.
Phillip Landmeier If I did that, my kids would kill me. Just watching videos can destroy everyone else's ability to just surf the web. Not the best internet... 5mb.
ReplyDeleteIndeed, my upstream is only 8M. Actually that's more than I thought. I'll run the fetcher when i can.
ReplyDeleteEd S Maybe I'll do it at night... but I would hate to have my ISP (own by the government) decide I was abusing it and turn it off. It would take months to get it back on.
ReplyDeleteFWIW, the bandwidth of the Warrior can be limited: https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior#The_warrior_is_eating_all_my_bandwidth.21
ReplyDeletearchiveteam.org - ArchiveTeam Warrior - Archiveteam
I never use mine for video. Mine's fairly fast for the USA -- slow as molasses compared to Europe or Asia, though. I normally get around 25 Mbps down and 2 Mbps up. But, since I've been running Warrior, the ISP seems to have adapted and now I get 40 Mbps down and 8 Mbps up.
ReplyDeletePhillip Landmeier But, since I've been running Warrior, the ISP seems to have adapted and now I get 40 Mbps down and 8 Mbps up.
ReplyDelete#JawDrop
John Lewis Hmm. It's not that fast. My friends in Europe were at that speed in 2003, even in small towns in Spain, and they were laughing at me. Now they get gigabit speed for the same price I pay here -- about 20 bucks a month and no data caps and they're still laughing at me. Even in a backwater like Albania, the slowest service there is 60 Mbps. Until a year ago here, I was paying $100 a month for the fastest service I could get here in West Virginia, which was 24 Mbits.
ReplyDelete$ speedtest-cli
ReplyDeleteRetrieving speedtest.net configuration...
Testing from Telenor Norge AS...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Trollfjord Broadband AS (Stokmarknes) [144.05 km]: 40.87 ms
Testing download speed................................................................................
Download: 39.77 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.27 Mbit/s
That's what I get here in northern Norway for a ~50 euro monthly Internet/TV cable subscription.
Though hopefully in 1-2 years we can get fiber for a similar price.
speedtest.net - Speedtest by Ookla - The Global Broadband Speed Test
Phillip Landmeier You might want to match your Warrior bitrate to your upload speed, as otherwise the bits pile up.
ReplyDeleteEdward Morbius Yeah, I've been watching since I began to see if that happened and the balance seems pretty good, either by chance or because Warrior throttles itself. It appears that data is downloaded and then compressed before uploading. I notice that when a download is complete, all four cores spring into action and clock speed jumps to 2.6 GHz for some seconds before uploading begins and CPU load drops back down. Of course, this is all indirect observation. Disk space never goes above 5 GB. I'm only running two instances of Warrior and letting them take whatever resources they want. I hesitate to tinker with it.
ReplyDeleteFilip H.F. Slagter Interesting. I have a couple of friends in Norway. They get much higher speeds than that at home but they're both in Oslo. Your connection sounds very similar to mine and I'm also in a small city (16,000) here in West Virginia.
ReplyDeleteHmm. The Warrior tracker appears to have commanded all Warriors to stop downloading from G+. Or, possibly, Google has found a way to block us. Dunno what's going on.
ReplyDeletePhillip Landmeier manual individual save requests (https://web.archive.org/save/https://plus.google.com/107049823915731374389/posts/1kQS3fvrceA) are still working, so it's not a sidewide robots.txt disallow rule fortunately.
ReplyDeleteFilip H.F. Slagter plus.google.com - I wonder what happened. Looks like Warrior operations halted at around 0200 U...
ReplyDeleteAll's well now it seems. Everything is running at full speed again and superstar Fusl is back in the game.
ReplyDeleteThink Google had a hiccup. Also, Facebook isn't looking so good either. Concentrated attack?
ReplyDeleteJohn Lewis I'm still seeking info, but I believe now that it wasn't Google's "fault". I think the Archive itself or the Tracker was overloaded in some manner. The top producers, who do this archiving seriously, have set up another target for uploads that can handle the data rate. I'm eavesdropping on their chatter. A few minutes ago, they brought up a target (server farm) in London and the top players directed their output there for later upload to the Archive at a slower rate. A few minutes ago, that target was accepting data at 600 megabytes per second. (bytes, not bits).
ReplyDeleteIn any case, we need more people doing it. We need to roughly double the number of work units per minute and keep it up to finish the job in the 19 days we have left.
ReplyDeletePhillip Landmeier Ok, posted to my person stream.
ReplyDeleteJohn Lewis Thank you.
ReplyDeleteYeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/-jyf6prCz7KFh5bypUVRoRC0dKiQTdRP3jjvVhOl8-QRZuTpzR58WNtfjvERybtM54Dew4pJW79rm6vxJiU7vOEkUsRRpNPoz0M=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/NC9g9PFaxb6entZf0F9mS9l8cuBF7H6sgIUj4FjtYHMhYVefZPlqohwwG0Q8IybHSQLHF1r35iFQnB5tUVIQuklAd0cAtrwdcQ0=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/vTCNZG5MSJI1fUEMUOKDh316sPC5MXTny92M7V7XTHXxRxt4c1s5SxNKVDtXe9mwHWQBZIpOGs_GfwGwioQ5JUqZLfLMrwC15KI=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/ocBnc9L1W4gxKbIdrIEocG5DwdAv_aYZr2R0y1hafaNVa-INi6DFjnN-2WsCR8LduBNyleljYRJr7jnQuytH4SFjN7yLdhI9jOo=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/qYHRF4QmAOpGfI5pgx_XeQ3YZ66PAg55fwucG2tBbtrqS8_aAKMGJ8C8hSAgGoYTWxcUQGG77xq3K-STvFYosP9THxZdX3x5ljg=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/TJ3eWIGxSO5axa5l81bR5IQ3_fO2zU8RKuOeoyl9kjlIVJ731XsYzJgisao9GcGc7F8vyQsv4yjgIM8cOCPQbclnA4QaZbidvxw=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/Mg662No0jTr5VpJ-BFMpf0qyDXUVfj_yjM16AFtD-LtaeO02nkMBgc6wT35yZJSSKyP2FWQ76aEGdfwvZE3g6e1WRONAmglCQzY=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/S5Mo51R4WmqMhCzDf2a7mK4ZXBcwekNH4JCHaZtx9jefqC6X0BE3YMAmf3gSbmWx0l7-vVU0-wUVlemXEfy08NKnEBqJJEmCetg=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/S99lsRuOquPnkjQzna3nYtza2ec6f6KiWeu8lhNH4UvdA-S4240n4Men9NPVV0EHnBIlEksJlxvLdxJyXifmEhQfooRJ8BEPcz8=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/31YxBIKrl4eg6ye0nhBHjNk2pskrERy_XV7iPVJvxwz5cJ9GzPwpgti6FvsXuQolOt8C7AWmgcpdw0c9oujXpFIjwMeK4_adEMo=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/N135ONNPOc2iu_8rCZaSW0-0ClgDyjQrb2lccv034Q-rLyyiS8GBfxMlQ_Gpvw_FYoTyNfU-QTrXeSl4jWi1SktV2-3mefmu97w=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/BHLVg-3aIACcZ-5yzr14nRdmIm-SBjw2-eRIte0MSaMsP1J8NellkSsq356kzmdljuTCIgo1b-Ep_wQW-t_Dfqurx8wurBcdx14=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/8v-SOAyDSCQX8VYDCcCfNs4fLfTFemRgX-RL588L_Wmi2sEXgfEzIfabpwCtEVzW7BnCVxJif4zsYz6prl7C8sKHvruAp6s96h4=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/fZgSJp3H-QyCnpEMXOjflA8ZzV-kQDfgcbnmlqRrSxbPzWxBQ80LK7lBHogV9qpotIjIRJnXc2WPfiaRbdTz-ZtUo5PGpS3Vloo=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/-tgsQaExOnPF9sx0UrPCOq0rS3aDdetfjQ-bANN3315Q4PbH2vGfgM7-SPABX3d6SgWYQTHrG8C5npGBX1pwS9tcLvHvVBBh1vQ=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/aHrOJsntA-qBx992dMvHclmPWMgwWiZsdd-7eB3ZkmO324dWSpyQpTweXDTAEvuHvQ4Jk-2Dx4RhegVLDiKDEHQinuCC6aaXbRk=s0
Yeah, the "large target" in London is making quite a difference, as you can see in the graph below. There's talk of adding another large target.
ReplyDeletehttps://lh3.googleusercontent.com/hA_EOsnHNFe775QmSBXQ-u33U1DQu078xdsziKwR5o29j4nzATgLAvDtX9Lik-AgC88Yxm1qRzh8UzAB7B3egsY8XAyRix3O-Yo=s0
I noticed one of my threads had been downloading the same job for over 8 hours, so I killed and restarted. It was a huge profile posting unremarkable images. I'm running 6 threads now. I'd really like to see it always with at least one download and one upload active at all times, but that's not quite happening even at 6. Might increase...
ReplyDeleteEd S Yeah, I see you're cranking it now! :-) Well, somebody's going to have to do that work unit. I've got one running now that's been downloading for almost five hours. Gigs and gigs. I didn't realize that some people post astronomical amounts of stuff. Kind of ridiculous. Do you really have that much important stuff to say? Lmao.
ReplyDeleteI've considered aborting it, but somebody's going to have to do it and gigabytes is gigabytes. It doesn't help my work unit score but it pushes my gigs up. So, Ima let it run unless I run out of disk space.
My thinking was that it's better for someone with massive bandwidth to pick up the monster account. It was possible an auto-posting bot, so only of interest to a social historian. And if I only had a couple of threads, and they were both stuck on such a mega-poster, that would mean less progress.
ReplyDeleteLooks to me like you've got a fat pipe and plenty of threads!
Over five hours now. I took a look and it's a guy with 500 followers who posts links to news stories, mostly crime news -- non stop since 2013.
ReplyDeleteI think I'd abandon that one! (I'd even wonder, if a work unit is booked out for completion and not done in 5 hours, will it expire? Will it even be accepted when it finishes?)
ReplyDeleteEd S Downloading seems to average around 20 Mbps. I run three threads, but have shut two down so this monster work unit can finish faster.
ReplyDeletePhillip Landmeier Some used Google+ as a kind of Flickr. Those are the hardest.
ReplyDeleteYeah. Still going. Over 26,000 posts now. All he did was surf news sites and repost links, so each post includes a photo. He never says a word.
ReplyDeleteLow value! I did a soft shutdown, waited for all other threads to finish, then did a hard shutdown. I recommend rather more threads, BTW. You want to be limited by your network, so you want both up and down traffic always to be busy.
ReplyDeleteEd S Exactly. That's mostly what I get with three threads. In this case, I need to shutdown for some maintenance, then restart, so I want this one WU to finish. 29,000 posts and counting, same dude. Lol.
ReplyDeleteEd S Yeah, I don't know what the timeout period is set to. I suppose I could ask. The timeout is probably days because they surely know there are WUs like this one that are ginormous.
ReplyDeleteI'm quite surprised that with only 3 threads you always have something uploading - even with 6 or 7 I often have them all downloading which represents a waste of my upstream bandwidth.
ReplyDeleteEd S I would run more if I had more disk space. I could only spare 15 GB for this. I'm thinking about adding an external drive and directing VirtualBox to use that, if that's possible. I plan to find out as soon as this damned WU finishes. 7 hours and over 34,000 posts -- still climbing.
ReplyDeleteDespite the waste, I'm considering aborting it. I was chatting on IRC and learned that unfinished or abandoned work units essentially go to the back of the queue. If my assumptions are correct, that means that this beast will get handled last. That means that if we run out of time at the end of the month, it will be one of the WUs that doesn't get done -- which would be fine. It's a big waste of space.
Back of the queue is quite a good idea! I would abandon. My instance only got up to 18G with 6 or 7 threads - I think there's not too much disk use. I haven't got very much more than that anyway.
ReplyDeleteEd S Yeah, I killed it. Then did a complete do-over. Removed the Warrior and files. Rebooted. Configured another SSD. Configured Vbox to use the big SSD. Unthrottled my CPU -- I normally keep it throttled down. Installed Warrior again and set it up for six threads. So now everything is maxed here and just starting to run again.
ReplyDeletegreat! Do you get to keep your username?
ReplyDeleteInteresting. It's actually using the additional CPU power...aaaand the cooling system kicked on. Hahaha. I wonder what it's doing with all that compute power. Strange.
ReplyDeleteEd S Hmm. I assume so. Haven't checked.
ReplyDeletePhillip Landmeier Yup.
ReplyDeleteWhat on Earth does it need the compute power for? It's got all four cores pegged at 3.07 GHz and never lets go. I wonder if restricting it as I was before was a bad thing. Maybe it does better or more intense compression if it has enough compute power available. If so, great.
ReplyDeleteFrom the FAQ:
ReplyDelete"The item I'm working on is downloading thousands of URLs and it's taking hours.
Please notify us in the appropriate IRC channel. You may need to reboot the warrior."
https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior#The_item_I.27m_working_on_is_downloading_thousands_of_URLs_and_it.27s_taking_hours.
this one was downloading for 8h with 36k items
ReplyDeleteplus.google.com - 溫寶麗
Ed S That is epic. It's all photos and none hers as far as I can tell. One category is "動漫美圖" or "Anime Beautiful".
ReplyDeletehttps://plus.google.com/100899512816358507414/posts/UbtJ6QyMWVp
ReplyDeleteI title this "Pull yourself up by the hem of your their pants."
(Oddly enough I couldn't post this comment when including the link preview...)
I've seen single uploads of 6 GB go by on the Tracker.
ReplyDeleteMy whole account fits in 360 MB on Takeout. lol
Got another one of these big honkers in progress. It's a self-promotion account with a bazillion posts and 12 followers. Lmao.
ReplyDeleteWhew, after three hours of downloading, it's finally uploading. I'm still amused by the user -- a woman who thinks she's an expert on having college-aged kids. 8,500 posts about it and 12 followers. I'm glad I'm not her kid. Yeesh.
ReplyDeleteWhy archive so much data that appears to have minimal value?
ReplyDeleteBecause it's a cultural record, of online life in a certain period. It will show what people were talking about, and how they talked about it, and how they connected, and how they failed to connect.
ReplyDeleteLike the menu from the Titanic, it's interesting once time has passed, but if you don't preserve it, you never get to that point.
But also there was a lot of excellent discussion. If 99% of posts were not excellent, we're most likely to be seeing those. The only way to deal with this in a short time frame is to save everything, and not make any judgements about what's important, relevant, or interesting.
Besides, what is irrelevant to us, might be relevant to others. :)
ReplyDeleteIn 500 years or 1000 years, this kind of information will actually be valuable if for nothing else than showing the network effect of certain cultural connections that we, living in it, can't even fathom yet.
ReplyDeleteIf something like G+ had been around when my late relatives were still alive and young, and their posts had been archived, I would've been very interested too in seeing what they would've posted about.
ReplyDeleteAlas, I have only a few scribbled notes and pictures without much context instead of them.
Letters and diaries I have from my mother and grandmother - lots to read thru. But handwriting is not so easy to read once you are used to tweaking this virtual interface so it is easiest to read.
ReplyDeleteI have copypasta a select few G+ posts from my profile, as I deleted.
Diana Studer Use a scanner to scan your mother and grandmother's letters. One day AI will read them flawlessly.
ReplyDeleteJohn Lewis (first I need a scanner)
ReplyDeleteThe Captcha that irritated people so much when TRYing to comment on blogs - was sourced from documents that AI couldn't read. Using (unwilling) human eyes to resolve the ambiguities.
Diana Studer And the game of Go was thought to be beyond the reach of AI only a decade ago. It now easily defeated the worlds best players.
ReplyDeleteThe same with poker.
Do I actually have to bring up Rock/Paper/Scissors? It's not random, you lose overtime against the AI.
What you think you know about AI will be wrong and sooner than you think.
afiniti.com - Rock, Paper, Scissors