Skip to main content

API/Scraping Question: Any way to retrieve a list of members for a Community?

API/Scraping Question: Any way to retrieve a list of members for a Community?
This is mostly for my fellow developers: is there an easy way to get a list of all members of any Community?

I know that Community owners can get a list through Google Takeout, but I'm interested in getting such a list for communities of which I'm merely a member, rather than an owner.
Preferably through one of the APIs, though scraping would also suffice.

One of the reasons for this is that I'd like to be able to have my script generate a list of alternate social media profiles based on the (public) links in users' Google profiles.

For instance, specifically for this community it would be interesting to see how many users have set their MeWe/Medium/Blogger/Mastodon/etc urls in their profile yet, and which platforms are the most popular based on that.

Based on my own Google+ Circles Takeout JSON files, here's an example list of most listed platforms in the users' profile links section:

Found 506 platforms:
— Twitter (132 items)
— Facebook (88 items)
— LinkedIn (57 items / 55 unique items)
— YouTube (58 items / 54 unique items)
— Blogger (48 items / 44 unique items)
— PicasaWeb (37 items)
— Flickr (37 items / 36 unique items)
— Google+ (30 items)
— Google Reader (22 items)
— Instagram (19 items)
— GitHub (19 items / 18 unique items)
— Wordpress (17 items / 16 unique items)
— Quora (14 items)
— LastFM (12 items)
— Pinterest (11 items)
— Tumblr (9 items)
— Delicious (9 items)
— 500px (8 items)
— SoundCloud (7 items)
— Wikipedia (6 items)
— Vimeo (6 items)
— Diaspora* (5 items)
— SlideShare (5 items / 4 unique items)
Live.com Profile (4 items)
— Digg (4 items)
— DeviantArt (4 items)
— Patreon (3 items)
— Medium (3 items)
— LiveJournal (3 items)
— Friendfeed (3 items)
— Foursquare (3 items)
— Cafepress (3 items)
— AboutMe (3 items)
— Orkut (3 items / 2 unique items)
— MySpace (2 items)
— StumbleUpon (2 items)
— SourceForge (2 items)
— Reddit (2 items)
— Pandora (2 items)
— NetVibes (2 items)
— Keybase (2 items)
— Hyves (2 items)
— GoogleCode (2 items)
— Goodreads (2 items)
— Ello (2 items)
— Dreamwidth (2 items)
— Dopplr (2 items)
— Disqus (2 items)
— Bandcamp (2 items)

The 'Diaspora*' entry is based on a list of most popular Diaspora* pods from the public instances list.
Obvious personal domains, or product pages such as Google's support documents, are excluded.

Surprisingly, none of the people I have in my Circles, apparently have their MeWe URL in their Google profile. It also surprises me that I have so few listings for Diaspora*.

Then again, based on the high amount of dead services still included, such as PicasaWeb, Google Reader and Delicious, I'm guessing that a lot of people (me included) have only bothered to set up the links section when they set up their profiles, and haven't kept it up to date since.

I'm still refining the output and data gathering for this analysis feature of my Plexodus-Tools script (https://github.com/FiXato), but I'll hopefully have this committed and pushed to the repository in a few days so other devs can start playing with it as well. :)

Anyway, to re-iterate my question:
Is there an easy way to get a list of all members of any Community?

Comments

  1. I guess one option would be to first work on my script to archive all the posts in a Community, so I can get a list of all users with some form of activity in a community, whether it is from posting a topic, a comment or a plus-one.

    In the meantime, to get a more expansive list, I guess I can also build a list of users who've interacted on any of my posts from my Google+ Stream Takeout archive.

    ReplyDelete
  2. I think a lot of people use/d their different platforms to have semi-different identities, particularly at the point where G+ 'had' to have 'real names' for a while.

    ReplyDelete
  3. One thing I could look at, is the AJAX calls the Community web interface makes to fill the members list..

    ReplyDelete
  4. Filip H.F. Slagter This is beginning to feel like a database built from spidering outwards from yourself and from interesting communities. For each likely source, get a list of posts, extract all the actors, get their about data. I can imagine running up against the API call limits.

    I would say that if somebody has never posted, commented or +1ed anything in a community then they're not interesting.

    There's never been any direct support for communities in the API, so that's not going to help. And the only other official source is Takeout and that's only available to owners[1] and has limited data.

    [1]Can Moderators of a community get the Takeout for that community or is it only owners?

    ReplyDelete
  5. This is also feels a lot like FOAF and its bringing back memories of microformats. The next stage is to start looking for auto-discovery headers in personal web pages for pointers to a YASN-Roll. eg


    Except that no matter how many times the microformat and Social/Semantic web people tried to make this happen, it ran into a wall. rel='me' has got quite a lot of traction. But the only other auto-discovery convention that ever really got used was for finding RSS/Atom feeds.

    This comes back to #SignalFlare and making yourself easy to find. It's great that I can manually walk round and follow links and have a chance of finding people's home profile on each new platform. But it's laborious. What I actually want is the bridge.joinmastodon.org - Find your Twitter friends on Mastodon extended for more source and sink platforms. eg on Diaspora, find all my friends from G+

    ReplyDelete
  6. Julian Bond I never trust services with access to my (e-mail) contacts and such though... it tends to provide them with more details then I'm willing to give them.
    Ideally I'd just give them a newline separated list of Google Plus profile URLs instead.

    ReplyDelete
  7. In a previous life as a YASN-Admin, we used to encourage people to upload VCF files and lists of email addresses and offer to mass invite their friends. And some of the platforms allowed you to download lists of your friends or directly access them. I guess those times are over! ;) There used to be a route to consolidate all the email addresses of all your friends (including Facebook) via Yahoomail but that disappeared around 2012.

    What I'm edging towards though is something like a local Thunderbird Contacts database but which has extensions to hold profile ID/URLs for multiple YASNs. And then a set of tools to populate it from multiple sources. And this brings us round and back to what the hell happens to Google+ Profiles, Google Profiles and Google Contacts post the G+Sunset. If Circles, G+Profiles and AboutMe get merged into Contacts there's much less need to build an external database. But we still need the functions to track down our contacts on other platforms.

    ReplyDelete
  8. Julian Bond Does a question about inding G+ contacts post-Sunset need to be added to the Questions for Google list?

    How should we frame that?

    ReplyDelete
  9. Edward Morbius here's a draft suggestion:

    What happens to Google+ Profile URLs and their data after Google+ Sunset?

    The Google+ Circles Takeout currently contains references to Google+ Profile pages on the plus.google.com domain, and apart from the name, contains not much else.
    What will happen to these plus.google.com/$userid URLs after Google+ sunset?

    Will profiles.google.com/$userid stop redirecting to plus.google.com and be used as primary Google profile information page? Or will there no longer be a way to look at the (public) profiles of Google users?

    Also, since some of us have submitted quite some useful information to our profiles (workplaces history (organizations), places lived, about me, urls (website, other profile, and other), skills, bragging rights, etc), which can still be updated from aboutme.google.com:
    What will happen to the profile data after Google+ sunset?
    Will this be available on the profile.google.com/$userid and/or aboutme.google.com/$userid URL(s) instead? Or will it be lost forever?

    (I AM NOT GOING TO MISS GOOGLE+'s BROKEN FORMATTING!) Seriously... The first attempts completely changed the formatting in ways I don't even understand...

    ReplyDelete
  10. Will Google provide tools for copying G+Circle contacts to Google Contacts? The old Contacts UI used to have a section for Circles but the new one doesn't. This presumably means that Contacts will no longer have explicit entries for G+ contacts.

    Contact displays for a single contact in Google Contacts where there is a connected G+ Profile have a quite significantly enhanced display with quite a lot of information extracted and reformatted within Contacts. Will this continue?

    ReplyDelete
  11. Filip H.F. Slagter Nesting bold and italic elements, in and order, breaks G+ on re-editing. #ShitIWontMissWhenThisHeapDies

    ReplyDelete
  12. Julian Bond I read a while ago that importing the G+ Contacts Takeout .vcf files in Google Contacts would automatically import data from their Google+ profiles into Google Contacts.

    ReplyDelete
  13. Edward Morbius the most stupid part is that editing and saving a comment without making any changes, can completely change the formatting... Which probably means that when you click edit, they try to convert their internal HTML back into their 'adoc'-inspired format, rather than using the original text you submitted last time.
    guano-psychotic, I tell ya.

    ReplyDelete
  14. Julian Bond (offtopic: weird, I thought I was already following you. Anyway, fixed.)

    ReplyDelete
  15. Your 'items' vary hugely in size (and therefore in their use or value)
    I will skip single photos, or tweets, in favour of substantial content.
    Blog post, or similar on LinkedIn or Medium.

    Also I see a few pioneers on G+ who have put their find me here in their banner. And MANY who have abandoned G+ with dead links to other dead sites / profiles.

    ReplyDelete
  16. Thanks for the mention Mike Noyes.

    We had a scrapper to get the number of members in a public community, but no scrapper to get the member of the community. There was/is no API for this and as far as I remember you couldn't get the full list even via the browser/UI. You can scroll through the list of members and as soon as you have reached the end of the list, additional members are loaded, but after a specific number of additional members you won't get more members to see (even if there are more available).

    ReplyDelete
  17. (Even with Takeout I see only a subset - many hundreds - of the thousands in my community.)

    ReplyDelete
  18. Ed S that sounds like a bug that needs reporting at least. I don't have a sufficiently large community myself, or else I would've done so.

    ReplyDelete
  19. CircleCount That sounds pretty much like my communities scraper -- "members" is given for public communities, though not private ones. I don't see a ready way to scrape individual member lists.

    Were you working off Sitelists for the full list of communities, or did you have some other way of prioritising what you looked at?

    Have you seen my recent sampling-based assessments of G+ Community sizes and distribution?

    7.9m communities
    mode size: 1
    median size: 2
    mean size: 128
    95%ile size: 118.5

    ... etc.

    https://old.reddit.com/r/plexodus/comments/9zx67d/google_communities_membership_analysis_preview/

    NB: I updated your April 2016 top-200 list (via Internet Archive), and found that 69 of the 200 communities included no longer exist. That's going by Community ID, not name -- there were another 116 name changes.

    https://plus.google.com/104092656004159577193/posts/3Sna8JoTjyA

    ReplyDelete
  20. Filip H.F. Slagter G+ Takeout -> Contacts. I suspect it was me talking about it!
    plus.google.com - Circles Takeout Export to Google Contacts Import. Choose Takeout G+Circles, ...

    I'm really not sure now what the hell happens. And it's hard to go back and try again, now that the people have been copied across. I * think * adding Takeout VCF to Contacts does create new entries that use the link to G+ Profile to create an enhanced contact entry. But you may then have to de-dupe with contacts that were already there. Everything about this is confusing. Even more so when you have auto-sync going on with Gmail and a Phone's contacts. And 2 alternate UIs for Contacts. Where contacts.google.com doesn't seem to be linked from anywhere on desktop web.

    ReplyDelete
  21. Mike Noyes Hooray. Yet another UI to the same data & tools. Click through to G+ and you eventually end up at the same Takeout screen but limited to just G+ properties.
    https://takeout.google.com/settings/takeout/custom/plus_pages,circles,stream,plus_one

    ReplyDelete
  22. Julian Bond oooh, being able to list the appropriate sections in the URL is actually quite handy!

    ReplyDelete
  23. Filip H.F. Slagter Yes, that's something I plan on snagging for the next set of prompts / howtos.

    Julian Bond I think we want to add the Profile to that also.

    ReplyDelete

Post a Comment

New comments on this blog are moderated. If you do not have a Google identity, you are welcome to post anonymously. Your comments will appear here after they have been reviewed. Comments with vulgarity will be rejected.

”go"