Skip to main content

Is Google+ Exporter faster with TOR switched on or off?

Is Google+ Exporter faster with TOR switched on or off?
It's running 16 hours now and has so far counted 26,230 posts and 691,455 comments from my profile stream.

Comments

  1. Faster off because any app running is consuming memory.Anything that is running consumes memory and slows down primary tasks

    ReplyDelete
  2. I thought so, too. Thus had turned out off.

    ReplyDelete
  3. Gerhard Torges currently I suggest disabling Tor to make sure the posts download works. It's not faster or anything but more stable, definitely​.

    ReplyDelete
  4. Gerhard Torges Tor was a workaround to IP blacklisting by Google.

    If you're not getting blocked, you can disable it.

    If you are, re-enable.

    ReplyDelete
  5. Is 16 hours for 26,000 posts a reasonable time, Alois Bělaška?

    ReplyDelete
  6. Gerhard Torges hard to tell, is it approximately count you've expected or is it much more than you've expected?

    ReplyDelete
  7. It takes longer than I expected.
    But on the other hand, the post count is bigger than I expected, too.
    Would a faster PC help much?
    Or a faster internet connection?
    I have a Windows 7 Core i3 machine and 500 MB/s cable internet.

    ReplyDelete
  8. Gerhard Torges only lover rate limits on Google+ servers would help, unfortunatelly that will not happen.

    ReplyDelete
  9. Trying to access anything through TOR other than basic mainly text websites and you're gonna have a bad time.

    ReplyDelete
  10. Too bad, Alois Bělaška.
    How does the app work?
    Does it parse the Google+ HTML feed in the background?

    ReplyDelete
  11. Ich denke da läuft irgendwas gerade nicht ganz rund mit der Anwendung. Der hat bei mir 12h durchgewerkelt und war immer noch nicht fertig. Und die Anzahl der Posts und Comments war völlig unrealistisch hoch.

    ReplyDelete
  12. 26000 posts in 8 Jahren sind 10 pro Tag, das kann schon hinkommen.

    ReplyDelete
  13. I accidentally used "Download Everything" in the F+Me exporter. Turned out to grab the communities I am members of, including the G+ Help Community. Ran and ran and ran... Sigh.

    Tor was indeed a workaround to G+'s automatic analysis thinking G+ was being abused. I was one of the people that was happening to, and the Tor workaround worked for me.

    But eventually G+'s automated analysis started creating problems for Tor access too, and it turned out to be necessary to turn Tor off. Fortunately in the time since then, I have been able to download directly.

    The F+Me exporter is not especially fast. It does not do incremental updates (that is, only looking at posts newer than your last download), possibly under the assumption that someone might have added new comments to old posts. It does, however, keep track of which images you have downloaded, and knows not to download them again... at least if they were there under the same name (no detection of duplicate images.)

    ReplyDelete
  14. Gerhard Torges Ist er denn bei dem Stand jetzt fertig? Bei mir war er gestern bei über dem dreifachen und immer noch nicht fertig => unrealistisch.

    ReplyDelete
  15. Ich habe noch nicht nachgeschaut, Cyrill Kunze.

    ReplyDelete
  16. I told the Exporter to save my main profile.
    I hope this includes my posts, being them private or public, including those I posted to communities.

    ReplyDelete
  17. Das dauert, ein erster Export Community Daten von Funny Technology lief auch locker 24h oder länger. Profil war schneller. Bilder etc sind natürlich ein Problem

    ReplyDelete
  18. I think without Tor has been actually faster for me, though it's not speedy because, well, Google. I doubt that my internet speed is a limiting factor, so maybe this will be useful for others.

    Over 35K posts in 14 hours without Tor with 1.8.4, and it's doing more work, finding more images (almost 2700 more so far) than 1.8.3 did. I have around 45K more posts to download (based on the 1.8.3 refresh numbers) before I can start downloading new images and videos. At my current rate (looks like around 2500/hour) I have about 18 hours left before I can start downloading more images and videos.

    The image and video download has been much faster, and no new videos were discovered in the first 35K, so it's mostly images. Therefore downloading the newly-discovered images should be relatively quick. (Famous last words, I know.)

    ReplyDelete
  19. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/Ay8twiZceCrMBWxeiBIy1qTKEcHVLWW9Xq-0qHObgT15aAFkNhl4ds-FweQw1fTvoma_8FAkqthWxfraFqV0f5sSFSIgPFQD6xO_=s0

    ReplyDelete
  20. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/RsXUWfrbNycAMwzrzimPJ9lfYDsqlWHEghj90ciwkMiiZLIQ3eE8C5_YY75spwiWmdqaRXBuYYPEFKjuPRx2I4-EWSmqXeIO-qja=s0

    ReplyDelete
  21. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/I4l8PMY3hdbECvowntDoxf-_nbzfI2CmTK7y8g99fLkkKs0_zDotvAhXGN2SObDz7n_lzsD1Tb0GQUSQn6YcU0bWsIWr1TPzOv-n=s0

    ReplyDelete
  22. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/UpD8aLDou_0BpTBD9Ldum8VfVSia6roUewhTa5XvdfYWTQ_BhY5RicQOnuMMM8WG6cIvgdQT5srKylIHAiWZ-mFq3mfcfzcz80Zt=s0

    ReplyDelete
  23. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/Vq3YG3O9ccyTeVwfOBZ4ej09DOgW2ZXu_3OWon6HY-7wJuB2OFJgPj4scngNcfO-Hi6-UDlG4ubKW3WWT_JeHF4mSmDHdA6c5GQT=s0

    ReplyDelete
  24. Oh my …
    45,000 posts and counting.
    Still not finished after approx. 40 hours.
    What if I quit and update to 1.8.4 now? will it start all over again?

    https://lh3.googleusercontent.com/fhcARpw8pD4UT-k3qQISQBsVijSauCA9bA4JBTHxD6OPeq8btvM_KuctM4SuN_DuiqcU43ko3KSsFT822yq5OZm8iPOeK6z1q_VL=s0

    ReplyDelete
  25. Gerhard Torges yes, it will start again.

    ReplyDelete
  26. That's bad, Alois Bělaška.
    I wonder why there are so many posts.
    45,000 in 8 years, that's 15 posts every day.
    Oh, and while you're here:
    While counting it hasn't downloaded any content yet, right?

    ReplyDelete
  27. Not sure, Alois Bělaška?
    You made that thing! 😁

    ReplyDelete
  28. Gerhard Torges the problem is not everybody is served the same Google+ web application. Impossible to fine tune the app for all G+ versions.

    ReplyDelete
  29. "While counting" it is downloading all the content except images and videos. That is a count of downloaded data. As it goes, it records images and videos to download afterwards, which is much faster.

    ReplyDelete
  30. Thanks for the clarification, Alois Bělaška and Michael K Johnson.
    I think the best idea is to let it finish this run, then upgrade to 1.8.4 and try another one.

    ReplyDelete
  31. Gerhard Torges that is what I would do in your place. Make sure to download images and videos before running again with 1.8.4

    ReplyDelete
  32. Will do, Michael K Johnson.
    I still find it kind of strange that downloading the "text part" is so much slower than downloading images and videos.

    ReplyDelete
  33. Gerhard Torges for posts we have to slowly communicate with G+ servers, images and videos are downloaded from more powerful Google Image proxy servers.

    ReplyDelete
  34. Do you think things are running slowly because many people are using the Exporter right now?

    ReplyDelete
  35. Gerhard Torges my job is building cloud software that handles billions of events per day. This difference doesn't surprise me in the least. The images and videos are static stream files that have mostly already been copied around the world on content delivery networks designed to get lots of unchanging bits to you as fast as possible. The posts are from querying databases for lots of little objects that can change at any time and so have limited ability to be cached, and then assembling them into data in the format the part of the program that runs in your web browser (or on your phone) wants. That's not only slower, but also was designed to run at human usage scale, not as a lazy bug, but taking advantage of that characteristic to bring other benefits.

    ReplyDelete
  36. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/CKjvgWpHG2e6V6QEBgnVN_pHVNPEOjbvcvHs6YvHlkqyGoksQRpU-PGTILWx8Ze1ojPHGBh4luse-ml6uUzyi2Jr5DOaK6c1bQHX=s0

    ReplyDelete
  37. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/6p1CF-HGO4gOKg_On07hxPFRydLMQ0jdQaUReq_0VFN58tVgIA8SR5mO-ljYp9OzUuCKlkT-nyHHTFxu6wQrv_c1u_Gn9bGbGO9U=s0

    ReplyDelete
  38. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/RCWr90U-xghntdnLjoDh_pBgyVgVo3v-WEQm0aMtClHtNycCYcTgc0fm7y4fe414NciFpoJllRVTjEPJ_hKarn0qeCK3aVW0SsKG=s0

    ReplyDelete
  39. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/zmbCNfSz5iB5OmMp_kv0AzdHQnIzRqpO5DJTMo4tC1ZJF_cG0YPUZxAwkgBVTfunirV6LOLFVxlm4pXDQEu_U4X6NZE9LCDV3ana=s0

    ReplyDelete
  40. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/x3849RLX3uuM41q_MxEihE9gsr8swhQ8qOOTYtIpoVuUcKtYEVDjvghZ4bygyfEGyARR-K-mbOJfDvg6_6KsfIWeSYqr_it38xp3=s0

    ReplyDelete
  41. 49,000 and counting.
    Are these really all my posts?
    Unfortunately, Google doesn't provide means to see how many posts a user created.
    https://lh3.googleusercontent.com/ihjB46D-JB3s3nV0zUgij9_PM5y13H7WyN4FSAuT6n7aQClAil_4R6N9ocXBw9cPzmKPEh03cbbTjVIcjBV2sEnPM_KFtdIPuzFz=s0

    ReplyDelete
  42. Gerhard Torges if you do a takeout, you'll see what Google finds for your posts. I would agree that 49,000 seems like a lot. I rescued a bunch of maker communities and between those communities I have imported into their new home just over 50,000 so far...

    I would strongly suggest that besides using the exporter, do a JSON-formatted and an HTML-formatted takeout. HTML should give you something you can unpack and browse on your computer, and JSON is more likely to work with tools people develop later.

    "Belt and suspenders"

    ReplyDelete
  43. I did JSON takeouts on March 14 and 30, Michael K Johnson.
    Google said both times there are errors in the G+ stream part.
    Nonetheless I'd like to check the number of posts.
    Do you have any clue where I should look?

    ReplyDelete
  44. Gerhard Torges I glanced through your profile. At least recently, you reshare as aggressively as some people retweet. All those reshares are included in your posts. If you have been as active historically as recently, that number isn't obviously wrong.

    ReplyDelete
  45. "At my current rate (looks like around 2500/hour) I have about 18 hours left before I can start downloading more images and videos."

    It has slowed down for me. I downloaded another 30,000 or so in about 16 hours, and I have about 15,000 left to download before I can download new images (6116 more found so far) and videos (2 more found so far).

    ReplyDelete
  46. When it comes to "belt and suspenders" github.com - FiXato/Plexodus-Tools has tools meant to work with the takeout data.

    ReplyDelete
  47. Maybe Google is already pulling off servers from the net.

    ReplyDelete
  48. Gerhard Torges Please DO NOT spread unsubstantiated rumours through idle speculation.

    ReplyDelete
  49. It wasn't meant that way, Edward Morbius.
    My Download counter is now at 62,000 posts for my main (this) profile and still counting.
    This is extremely fishy.

    ReplyDelete
  50. Alois Bělaška does 1.8.5 have anything relevant here?

    ReplyDelete
  51. 1.8.5 may help. I’ve fixed a few bugs there but no guarantee.

    ReplyDelete
  52. I do hope so.
    Should have started earlier.

    ReplyDelete
  53. Does it scan the profile from new to old posts, Alois Bělaška?

    ReplyDelete
  54. Gerhard Torges it scans the feed as you see it on the web, from newest posts to the old ones.

    ReplyDelete
  55. Gerhard Torges by using G+ web application api.

    ReplyDelete
  56. I started over with 1.8.5.
    Just at roughly 3,000 posts now. 😕

    ReplyDelete

Post a Comment

New comments on this blog are moderated. If you do not have a Google identity, you are welcome to post anonymously. Your comments will appear here after they have been reviewed. Comments with vulgarity will be rejected.

”go"