VCard Format and Google Takeout
Just wanted to share a little tip, based on a mistake I made. When you are downloading Circles or Contacts using Google Takeout, you have the option to select CSV or VCF format. CSV is going to be more useful down the road, but the one thing that VCF is good for is importing into mail and address book clients. If you go the VCF route, you will see a bunch of files, each named after a G+ Circle (or if you are exporting from Contacts, it will be the various labels you might have used). It will look like a mistake and that Takeout didn't actually export the contacts. But all you have to do is double-click on that file and your address book should prompt you to import all the contacts for that specific Circle or Label.
For example, I'm on a Mac and the below image was for my "Active Engagers" Circle here on G+. When I double click on it, the Apple Contacts app opens up and prompts to import the individual contacts for that circle.
I at first thought that the export was flawed because the files were so small and the individual contacts are obvious. They are grouped and need to be opened up by the contact manager.
Also, always get the JSON checkout as well. It will likely be the defacto standard later for importing. (Programmers love JSON.)
ReplyDeleteYes, John Lewis, and the JSON format is going to be particularly important for people's posts.
ReplyDeleteJohn Lewis JSON does not apply to Contacts exports -- those are VCard (VCF) and CSV if I recall correctly, and as Gideon Rosenblatt's noted.
ReplyDeleteEdward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/TWW4SRt9ldmE-CnRrdVBrSd7iGUdngsawdvpbWhlWeJWHi0wrNAYQoReJTznlLfx5MH_byYrQL1eicZCyWpH0l69GotIheoEX99U=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/kbsxTliA5HcR5idvRMrsYXg_5DB7LYlbnGxZ2mew22vdPGPcNu0AKZvF1JNQUINQTUfZTcgTap7Mubvm70KINzkHHQ-FfWjCuSrZ=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/TUoqdb53yS_cUjTHXP9X4j3LiXtgqw6fKnGn0N9NOm7YvJyj2UDQocvItWIpzHZuiED2CyuKk7ABPDTjXxP2-2zpjs-wL2sPHSqO=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/FW3Sj83CG5ryI1Nr19SlxlkmC8DifJw8rxjrpAS_IGmpmp161nkyOl0W1GvHMNWY7QWwBcGxBG7h1JeiyPd71Y1DhMYuKEtTnZda=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/xerD6TRERgkERO0tTMNv0Ph7UzlzGJOTh8qTyvGhpmR5EzFeq3q0GN_5yzSNGh6LqigBhZOn_Fu5d6O3wM8XknuOrwMP-0ZPpyba=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/FDcAqUXwQuPpEExcnWKUxxXUTDTrSjNGpxzcKvRetFA0QkQaZkf6tHmzT7qb--mCf8fM9QPL89raJGc-KMDsazR0QA3VKINYb971=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/8YLXGLwuPHwnCABfV5YHylRWTtjgpjru_Q-yJouCRplseXqhEqxbr6T-NqFM-g9Tu30Uczjf-6zwPg7imrVobhHc2aQO2GZrudIC=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/NAn4FeYCH_kgySqHvaM18qIeU2bkDMerSkapHgXPd283LNHY1iMvn8wN8k79Ybj_NKTKNwe8ZrRhuBAGkuab6uDUACgJDn6BlX8r=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/y7r5QWXauFL4eZQBX8OHTA_zNMorqYpsllFul208IqX_p9kb3w7tqpsac9kUBamIQ0lcrPcjaQuUZPYrL6tZ_RzlsCFSh_fTexwJ=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/BojqCC3jNxJC-IdEbzrrEu5rZjTMAzZ9k74CDy5leEBrTktDs6YnsaD1XXhfyt9I-5nRYybQfYgR2w4FFyIPG_sPcxi0nNagpgRe=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/ReafjQJ_n_YBBqZ4ldvERlP__m4Pm_BWpYKufcnSNW_b0W3rU6DHOlL_Ui7VEoGH4QDqnnHOK1WKe8tiE_aLDuiLt2oAd2I7MtpV=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/cWmrDZM71ECodhl64k1y9xNu2p4PRQ0v_WfIiJazuGNKwhtZ4Nu54Qyb-okrc8bbQN4LEDQFMOx4AdU26J-5fT0h1CZsc2VwnTv9=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/n-oZrYbt91n5x6edN80GuzW8DI_7vL0HvzQ7qggfUNsf4OrKxXRKMCO755bousa6uw8oGjXUfwGSpX5shQzyHUNsZyyKLpDBP_Ul=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/fGIQ7c_wUvXO-aPm1_52c-njfhMV0bANAYsBIRKsJtHhe1cUU-_PLeu3xdzpDgfr2zgHxV2s-v9yAdLcHfooMFm3-mg8-au1OsoX=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/Sw5LbBrBCvjww8hRPT8Sz-3YBbhpfEUvfE6DcTOx9sH3c_G3EEzOzE3QHTivW_J7hNRKPRbYlATDZczxYwlh2eBfx3Ff7Q3vkfcg=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/iDGxefk-PZHpGV1VHZQNISmLJZ5tDFQ9Lqrji6OhT77lfp4D2sX6RQmcarEAlrzSXEtW33yMyY6R42yy2jRQUy3VTLo0OddfIMNZ=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/vtDb52p8F53V-cVC-cEWTvNmrIlEop5pWVW8EkEJAw-1sqkYh6gvJkx2i0a-i9ANQeiggUsIcfmXQP3P1E3US0FmjhVSQQUIKTMv=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/p09CTVmRnD9C0gJtK02jzuvxbwFPHBgMR22Q6jp8rdp1ckLIj4KPHWoZp0KgvPZGhnYJo-jUF-Q_ahYLsZScgp_0DlBxXTP7_S1v=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/yzADCnPQiy-4jHm_91sNicCXLi_scDlMGOJGYEhlQ3_YOktnNFTukFq8lps7FeUhtfYiucIlfbJ_Js7gDHLKdhKv44TF1B8M4zyA=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/KQ_e6fUg7X-EUQM7Gcait2EsDn8M1ZECa7a_GZjrhLiyLurH85EO2m0jRtfPlUHgqrh4BpjFPONf-Oi4tdOS-uxlcxT9e_muSmSO=s0
Edward Morbius, actually, that's what I recalled too, but just to be sure, I checked earlier today and JSON is an option.
ReplyDeleteI could swear this is new.
https://lh3.googleusercontent.com/5GniJFtkHUZ2T4CVFkk-uLx95diC8i6M66KaJeD7zwpHO10QX0KKxR-GQsUzKwPAZOk6oCGzh1AdGbowAtorU4Sekoyj-1JapJXw=s0
Gideon Rosenblatt Good to know, thanks.
ReplyDeleteYes. JSON is new.
ReplyDeleteThe VCF files (and the other formats) are small because there's very little in them.
BEGIN:VCARD
VERSION:4.0
FN:$fullName
N:$lastName;$firstName
NICKNAME:$nickname
URL:$googlePlusProfileUrl
END:VCARD
And that's it. I * think * but I'm not certain that if you import these into Google Contacts, it uses the Url and the G+API to lookup and get all the other public contact info. $googlePlusProfileUrl is sufficient to do this. At the moment, I'd recommend this route get as much as possible into Google.Contacts from available sources and then export it to some other contacts file.
Which suggests more code to write! Take the Takeout.G+Circles.JSON code and add all the data you can get from the G+Api.people.get call to produce a much bigger JSON contacts file.
ps. JSON option is broken. It actually produces the same old, same old, HTML. I guess it's a work in progress.
Julian Bond do you know when this showed up? The fact that we're seeing any changes here is actually important news, in and of itself.
ReplyDeleteGideon Rosenblatt I didn't notice it until you mentioned it today. My last test run on Circles was 22-Oct-2018.
ReplyDeleteah, so I didn't missselect... I was wondering why I was seeing html files when I was sure I selected json.
ReplyDeleteUpdate: Google+ Circles now actually exports to JSON. It no longer creates HTML files even when you've selected JSON.
ReplyDeleteUnfortunately it's still rather useless as it contains way too limited information:
{
"person": [{
"firstName": "FiXato",
"lastName": ".",
"displayName": "FiXato",
"nickname": "FiXato",
"profileUrl": "https://plus.google.com/108667492429930081201"
}],
"name": "CircleName"
}
Unfortunately for Google Contacts (not Circles) they still haven't fixed my reported bug with Group Labels that contain periods in them, not having the .vcf file extension for all the Contacts in that group.
ReplyDeleteFilip H.F. Slagter yeah, it seems like, right now at least, they're leaning toward the view that the only data that belongs to the end-user is the mere social tie itself and nothing about the person on the other side.
ReplyDeleteGideon Rosenblatt which is kinda weird, as you can set which of the data you want to share with others...
ReplyDeleteFilip H.F. Slagter At least with the profileUrl you can use that as an identification number to associate.
ReplyDeleteCircleCount Do you have a DB of profileUrls (The Google+ user IDs associated with each Google+ Account?)
On my list of code that needs writing.
ReplyDeleteTake the Takeout.G+Circles.json, extract the profile URLs, extract the user ID from each URL, use that to do a G+Api.people.get call to populate and then write out a much enhanced JSON/VCF file.
John Lewis CircleCount deleted all their data due to the GDPR.
ReplyDeleteJulian Bond That's actually been on my todo list too, lol.
ReplyDeleteFilip H.F. Slagter "NOOOOOOOO!" <-- my Luke Skywalker impression.
ReplyDeleteJohn Lewis gooooooood, embrace the dark side!
ReplyDeleteI could do my Darth Vader impression, but I need to get very very drunk to bury my head in my hands and breath heavily 😂
I can't remember if Beautiful Soup (Python) and deal with JSON. I'll have to look, but it's my very favorite way to parse marked up things. I'll check and report back.
ReplyDeleteAnd... no. BS4 doesn't natively extract from JSON.... it's built for XML/HTML constructs. Pity.
ReplyDeleteJohn Lewis no need to extra libraries. Python supports json natively, and allows you to load json files as dictionaries:
ReplyDeleterealpython.com - Working With JSON Data in Python – Real Python
https://docs.python.org/2/library/json.html
Got the API part of my Ruby script ready; just need to get the json aggregation done after dinner. :)
ReplyDeleteOnce I got the command line version ready (which requires setting up a Google Cloud Console dev account to get app credentials), I'll also see if I can make a web-based version of it; as long as it doesn't eat into my own Cloud quota.
John Lewis Google+ sitemaps included profiles as of 2015, I haven't looked at them since.
ReplyDelete50k files of 50k entries, 25 GB last I looked at them.
https://plus.google.com/robots.txt
Here:
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.gstatic.com/communities/sitemap/communities-sitemap.xml
Sitemap: http://www.gstatic.com/collections/sitemap/collections-sitemap.xml
Sitemap: http://www.gstatic.com/s2/oz/content/topics-sitemap.txt
Filip H.F. Slagter, this is the script for aggregating profile details from the API with the rudimentary Takeout details for Contacts and Circles?
ReplyDeleteInteresting: The Profiles sitemaps were last updated 2017-03-01.
ReplyDeleteEdward Morbius, wait, do you think that means there were only 50K users back then?
ReplyDeleteGideon Rosenblatt No. Each sitemap has ~50k entries.
ReplyDeleteRobots.txt > Sitemaps > 50k profiles sitemaps > ~50k profiles/file
Gideon Rosenblatt You can provide multiple Sitemap files, but each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 50MB (52,428,800 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 50MB. If you want to list more than 50,000 URLs, you must create multiple Sitemap files.
ReplyDeleteIf you do provide multiple Sitemaps, you should then list each Sitemap file in a Sitemap index file. Sitemap index files may not list more than 50,000 Sitemaps and must be no larger than 50MB (52,428,800 bytes) and can be compressed. You can have more than one Sitemap index file. The XML format of a Sitemap index file is very similar to the XML format of a Sitemap file.
https://www.sitemaps.org/protocol.html?x
sitemaps.org - - Protocol
Got working code to merge the Google+ REST API Person#get data into the Google+ Circles Takeout json. Just need to clean it up and write some documentation on how to actually use it.
ReplyDeleteHopefully I have some free time tomorrow to actually finish it up, so I can release it. :)
Advance Warning: For now it'll require a recent installation of Ruby (https://www.ruby-lang.org/en/downloads/), setting up a project on https://console.cloud.google.com/ to get a credentials file, and perhaps a bit of understanding of the commandline.
ruby-lang.org - Download Ruby
Well that's annoying. There are fields in the G+.people.get API docs that simply don't exist any more or are not delivered, even if the profile has them set to Public. So even if you work through adding API data to your circles download JSON, you'll never get any email addresses, for instance.
ReplyDeletehttps://developers.google.com/+/web/api/rest/latest/people
{
"kind": "plus#person",
"etag": etag,
"nickname": string,
"occupation": string,
"skills": string,
"birthday": string,
"gender": string,
"emails": [],
"urls": [ { "value": string, "type": string, "label": string } ],
"objectType": string,
"id": string,
"displayName": string,
"name": { "formatted": string, "familyName": string, "givenName": string, "middleName": string, "honorificPrefix": string, "honorificSuffix": string },
"tagline": string,
"braggingRights": string,
"aboutMe": string,
"relationshipStatus": string,
"url": string,
"image": { "url": string, },
"organizations": [ { "name": string, "department": string, "title": string, "type": string, "startDate": string, "endDate": string, "location": string, "description": string, "primary": boolean } ],
"placesLived": [ { "value": string, "primary": boolean } ],
"isPlusUser": boolean,
"language": string,
"ageRange": { "min": integer, "max": integer },
"plusOneCount": integer,
"circledByCount": integer,
"verified": boolean,
"cover": { "layout": string, "coverPhoto": { "url": string, "height": integer, "width": integer }, "coverInfo": { "topImageOffset": integer, "leftImageOffset": integer } },
"domain": string
}
Julian Bond Which is why we keep telling people to use Takeout often... it's going to change over the timeframe. This is an important lesson!
ReplyDeleteJohn Lewis it would help though if you could exclude the media files from the Google+ Stream takeout though, so you don't have to download 40GB+ every time...
ReplyDeleteJohn Lewis I suppose change is quite likely. But so far we've only seen two minor changes in the last year or so.
ReplyDelete- activitylog fixed, when it was broken
- Circles JSON added
And that's it.