Enhanced G+Communities Takeout is available
Hooray! The enhanced G+Communities Takeout is available with full content of Public communities available to Owners and Moderators. The Posts section is generating a single file for each post just as with g+Stream.posts and in the same format. Posts.JSON is correctly producing a JSON file for each post.
The one oddity is that Community.Summary is always in HTML even if you specify JSON.
https://takeout.google.com/settings/takeout/custom/plus_communities
Thank you Google, for delivering on the promise under the wire. It's still just early March!
If anyone successfully uses this for a big community, please report back.
#sub
ReplyDeleteMy net2o importer can deal with the new format, only one new tag was added. I had to make sure that all the duplicates really are perfectly deduplicated, and that the attribution signature is correct.
ReplyDeleteFortunately, the summary isn't needed; if you import all the postings, you get the summary.
What's missing is the category inside the community.
/sub
ReplyDeleteHow abt collections...
ReplyDeleteNasreen Malik Collections are part of your normal takeout.
ReplyDelete/sub
ReplyDeletefyi Mischa Krilov
ReplyDeleteNasreen Malik as Bernd Paysan indicated, Collections are indeed part of the Google+ Stream Takeout.
ReplyDeleteTo be precise, in your Google+ Stream Takeout archive you'll find a separate JSON file for each post you've made, in the Takeout/Google+ Stream/Posts folder.
In this JSON file there is an item named "postAcl" (where Acl stands for Access Control List), which contains one of several other Acl items that indicate what kind of audience setting(s) it had:
visibleToStandardAcl: the original basic visibility control that is used when a post is public to all, or limited to one or more specific circles, all your circles, extended circles, or specific people.
eventAcl: used for posts made within Events
communityAcl: used for posts made within Communities
collectionAcl: used for posts made to Collections
this last item also contains sub-items to indicate the resourceName (a sort of unique ID, the part of the URL which indicates which Collection it's part of), and the displayName.
So, if you want to collect all your posts that were made to a specific Collection, you'll have to find all the JSON files that have the right collectionAcl; my Plexodus-Tools has instructions on how to achieve that with the command-line tool `jq`, and the library of jq functions I wrote: https://github.com/FiXato/Plexodus-Tools/blob/master/README.md#examples
More details about the data structure in these Post.json files can be found at: https://github.com/FiXato/Plexodus-Tools/blob/master/activity_data_structure.md
github.com - FiXato/Plexodus-Tools
Nasreen Malik Your own Collections are included in your takeout, if you've included Streams data.
ReplyDeleteYou cannot directly archive other's Collections.
Is there a way to get the comments on the Community posts? If not, it's pretty much useless to me. The purpose of a community is the interaction.
ReplyDeleteSo if we do this must we learn how to convert JSON to HTML?Does this require a tech learning curve?
ReplyDeleteWi aM hEFF! ... good catch!
ReplyDeleteI hadn't actually noticed yet that the comments were missing from the Google+ Communities/$communityName/Posts/*.json files...
I agree with you that it's fairly essential for comments to actually be included in a Community export...
These are the top-level keys that I found in the G+MM community's JSON post files:
"activityId",
"album",
"author",
"collectionAttachment",
"communityAttachment",
"content",
"creationTime",
"link",
"location",
"media",
"poll",
"postAcl",
"resharedPost",
"resourceName",
"updateTime",
"url"
"comments" indeed seems to be missing :(
Filip H.F. Slagter GAH! Oh, come on Google. This is ridiculous.
ReplyDeleteI've sent in Feedback through the Send Feedback option in the vertical ellipsis (⋮) menu on https://takeout.google.com
ReplyDeleteIt might be a good idea for some others to leave similar, polite and constructive, feedback to show it's a missing feature more than just me cares about.
Likewise, I just found out that the comments (.primaryText) in Google+ Stream/ActivityLog/*.json (Comments, +1s on comments/posts and Poll Votes) don't include any kind of formatting; no HTML and even no newlines. This makes a lot of the more insightful/complex comments rather illegible, especially those with bold/italic formatting or lists.
Filip H.F. Slagter Julian Bond Wi aM hEFF! Yes, no comments section in my community takeout, either. Just the plain postings.
ReplyDeleteGoogle, WTF? I'll probably send them a GPDR request on March 31, with all the missing parts requested.
Leathur Rokk If you just want to quickly read the archive with minimal effort or post the whole directory somewhere, HTML format is good enough and requires very little work. If you want to use the contents to import it into some other system you probably need to start with JSON and then write code to do the import. However actually writing the code is left as a project for the student. There's not a great deal of help from anyone official although people here are building a body of code to help.
ReplyDeleteBernd Paysan not sure though if original formatting would fall under the requirements of GDPR... I have a feeling that even returning plain text would be sufficient for them.
ReplyDeleteFilip H.F. Slagter Machine readable in a common format. JSON is ok, but losing an important part of the contents is a real problem.
ReplyDeleteWe don't have the +1's of the comments in normal posts, either. The +1s are important if you have a lot of comments, and want to see the relevant ones or such.
No plusOnes either…
ReplyDeleteBernd Paysan do you mean your own +1s on comments? If so, there is Google+ Stream/ActivityLog/+1s on comments.json
ReplyDeleteIf you mean +1s on the posts in Google+ Communities/Posts/*.json, then that indeed also seems to be missing. No 'resharers', no 'replies' and no 'plusOners' are included in the JSON files for Community posts either...
Filip H.F. Slagter There are no +1s from others on comments in the normal stream takeout (so you can only extract your own +1s from the activities), and in the Communities/Posts no nothing.
ReplyDeleteBernd Paysan huh, you're right, I actually also hadn't noticed that before...
ReplyDeleteThe Google+ Comments API did include a .items[] .plusoners .totalItems count, but not a list of the actual people who gave it a plus one.
I guess the only way to retrieve that information now would be to scrape the pages.
Filip H.F. Slagter The takeout does not even include the "totalItems" field. There simply is no plusOners for any comment.
ReplyDeleteSo far, I think, all these takeouts were just more theoretical options, mandated by laws like GDPR, or before offered voluntarily, without anyone seriously using and analyzing them. Google+ is the first case where people take the takeouts serious. That's due to the large amount of nerdy contents here.
Just launched the archive process now on my communities, largest being linked below. Will report back when it's done. Edit: Done, 20 megs. Apparently I'm not very popular :P
ReplyDeleteThorium Now
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/9dDcYjHwHJLu6kkicI1p1X-Y7ihvJm0_oJeOf50gPMQVwAPn3r7ZskVaMGZCd78UHm2r6T4u9IDElRvtL2pbMCODNV8clVAq8mfv=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/mK7OO6vUvqlMkO5rx8Hnrama8IIjWDc4RLvMGVar_XpulDQk4UVyRv-wuKNfZupp1HawXi8ZIkKqsOGsT4Fpw4-G2733IY5EeXPo=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/c1KcYN9fdGC8G6oOCG_uKJsh7HPHfTSpmLk5f9i6WlP8o9U-PqwK9N4sSttZ4HJKc77vqvsl3XHciawCDRDOmLlYFvjVVmxIcCxr=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/C5y7Zp9faVSPxC9cLz9NNuN1IZ1GHhOwvJlzD7c0oXVkii1Th-Oh1kX5LJG-JKt0alnnUkLIzRSn-wvGCL2ouhPedFsHNe0fj9j5=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/ocvpLGQk0D3NsfA09oyQFg4jQPXWZ9kgqhf4XJzgrVV0BqgVGQD20zuWuF7iDF0qpQAhBLwlLpOJiCDK9Dupx3FJxafQn-vTh6r2=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/wY9pMATC4FiGZbQEA7p93P5kZf3I_bVXG8scDbsyjms4bMeJlh_-zVblqQ4EykOiOFM8n3lFqpbcSg1lYvFm1qLRoU6G4ZxR0yJq=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/IKdOpeDHMrr0f-12Y1FYIG_k9VGZ9wlcjwYmssQYxOVD6WRuV6Tt1Q6IdGY2Cl_aHYlgMli63l5cRmgr7_ZvkJTg2Dcg4CQLib-8=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/Vk0OmSR43zkg9XzNHxkze0IzYypfKY3c6ACbD7qeRVnO2IBVa9bocSeTjg3XChXEksEzu4veUYEAiIvzqM1aSXkW61HrmhYtbOQH=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/-Kw6gW9wGQl8ByM9tK13C6v1c2E_I1kYPHBCgSfZIIXclbJanX0slJNHkpuRp16WzNFwtTVl3LAgg2s0S4gqMJ3Th7u5vCNl2NGC=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/Fmj5k2hbaiAUIAu_CW8nSN9Q9OPhhOfpkG5XVK-bc86w8C7WxFKh47Oi4dQW4vU4M-idfoNg1bK4sOF6ZdReMmhh65j2ba2HhBx6=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/Ns0tqphRRre2-MjcJPFBiFEObIQzY2xS_NcK4ps0koEaDO-iEdII7IuJbdTPfkyMxGaZIVDLPs9gVt6vz_6PLnJoJm7W6W4f58xx=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/n7ssdLN8q_RRrkExtwFiWz1zlirZPMTR0gyTElwlYdjUHdd91cPVfXxXyr2OpaaJ8mPzRuhxcvJrmewsnG9bpLjLHfEh2b_FSr9z=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/hstAyBG0sMhkWoniuP9sgaI1c624GkohhWYPmNpa2mEmszsDudOOarR_TArIqlWJSJKJxulyRqo0cQ2-MEdhd0v-Wh6_0JyiaK-E=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/7yjXukGTG1824A9cT0YVcK4kiXBb2A5XfWHlrPjFYwXKEcsUMDdVWRk7aiW6O5WytiCviqoYxtDfaI-txkKFK23ocsCD0jtcv9NI=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/N5feOH9bBSnIuM1psY7nG4BMNiEGqeNDkLIEo-d1PBCFphB50XGq7TFygNj8shJ_UjGoRGE6gpP4137zNrLFdYrCB4BNxSxEtPcC=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/57UzWWeBfe19UjaXY8mKyv3p-jnGa7GOooJbYCNRoeuCGnaDSxxJX1crVytfK5eVenNHoyTtNaS8_kv4X-z2F6Swx27BAhcfsyFJ=s0
The Google export tool failed for me for Communities and Stream! What good is an export tool if it does not work?
ReplyDeletehttps://lh3.googleusercontent.com/DAluH-wR9diO4OVarzqskTLYsnOFmgPYkxPP-hWvjcmGOau3FIzbiKbGLVnpYspwm9i7RpQLjiJl9TLR4UBNu_V6JTnoLHJgwNbk=s0
Johnny Stork, MSc Look at the index.html, it will give you some more details.
ReplyDeleteIn general, you need to “warm up” the cache to get everything, so the first two or three takeouts won't be complete.
Johnny Stork, MSc as Bernd Paysan wrote (and as is mentioned in the red text), open the index.html file from the archive in your browser. It will list the files that are missing/incomplete, and often will include links that you can click to manually try to download them.
ReplyDeleteIt's likely some of the files are in 'cold storage', and thus need a bit longer to be retrieved than the Takeout process is set to use for retrieval.
Once those files have successfully been requested again, it's likely a next Takeout request will be more complete.
Bernd Paysan In general, you need to “warm up” the cache to get everything, so the first two or three takeouts won't be complete.
ReplyDeleteWhat complete bullshit! Not your comment, but that professionals in the biggest IT company the world has ever seen could design a system like that. "Run it down the hill again and let's see if the brakes fail again.", "Just kick it a couple of times and maybe it will work on the 3rd time".
Julian Bond Yes, Google+ is a mess behind the scenes...
ReplyDeleteJohnny Stork, MSc It does download but saves even pages u don't want and u need to hv enough spaceBrandon Sergent space in GDrive...
ReplyDeleteNasreen Malik
ReplyDeleteThanks :) I made another archive and it was again only like 21 megs, guess that's accurate /shrugs
Brandon Sergent Try copying collections using Coral draw...I hv managed to save my poems on Poemia...And a friend transferred all my poems on Collections on Coral draw...
ReplyDeleteThat's excellent, I'm glad you're not going to lose any work, pretty sure all my stuff is captured as well :) Sidenote: Fuck Google for closing gplus >:(
ReplyDelete