Is Google+ Exporter faster with TOR switched on or off?
It's running 16 hours now and has so far counted 26,230 posts and 691,455 comments from my profile stream.
It's running 16 hours now and has so far counted 26,230 posts and 691,455 comments from my profile stream.
Faster off because any app running is consuming memory.Anything that is running consumes memory and slows down primary tasks
ReplyDeleteI thought so, too. Thus had turned out off.
ReplyDeleteGerhard Torges currently I suggest disabling Tor to make sure the posts download works. It's not faster or anything but more stable, definitely.
ReplyDeleteGerhard Torges Tor was a workaround to IP blacklisting by Google.
ReplyDeleteIf you're not getting blocked, you can disable it.
If you are, re-enable.
Edward Morbius exactly.
ReplyDeleteIs 16 hours for 26,000 posts a reasonable time, Alois Bělaška?
ReplyDeleteGerhard Torges hard to tell, is it approximately count you've expected or is it much more than you've expected?
ReplyDeleteIt takes longer than I expected.
ReplyDeleteBut on the other hand, the post count is bigger than I expected, too.
Would a faster PC help much?
Or a faster internet connection?
I have a Windows 7 Core i3 machine and 500 MB/s cable internet.
Gerhard Torges only lover rate limits on Google+ servers would help, unfortunatelly that will not happen.
ReplyDeleteTrying to access anything through TOR other than basic mainly text websites and you're gonna have a bad time.
ReplyDeleteToo bad, Alois Bělaška.
ReplyDeleteHow does the app work?
Does it parse the Google+ HTML feed in the background?
Ich denke da läuft irgendwas gerade nicht ganz rund mit der Anwendung. Der hat bei mir 12h durchgewerkelt und war immer noch nicht fertig. Und die Anzahl der Posts und Comments war völlig unrealistisch hoch.
ReplyDelete26000 posts in 8 Jahren sind 10 pro Tag, das kann schon hinkommen.
ReplyDeleteI accidentally used "Download Everything" in the F+Me exporter. Turned out to grab the communities I am members of, including the G+ Help Community. Ran and ran and ran... Sigh.
ReplyDeleteTor was indeed a workaround to G+'s automatic analysis thinking G+ was being abused. I was one of the people that was happening to, and the Tor workaround worked for me.
But eventually G+'s automated analysis started creating problems for Tor access too, and it turned out to be necessary to turn Tor off. Fortunately in the time since then, I have been able to download directly.
The F+Me exporter is not especially fast. It does not do incremental updates (that is, only looking at posts newer than your last download), possibly under the assumption that someone might have added new comments to old posts. It does, however, keep track of which images you have downloaded, and knows not to download them again... at least if they were there under the same name (no detection of duplicate images.)
Gerhard Torges Ist er denn bei dem Stand jetzt fertig? Bei mir war er gestern bei über dem dreifachen und immer noch nicht fertig => unrealistisch.
ReplyDeleteIch habe noch nicht nachgeschaut, Cyrill Kunze.
ReplyDeleteI told the Exporter to save my main profile.
ReplyDeleteI hope this includes my posts, being them private or public, including those I posted to communities.
Das dauert, ein erster Export Community Daten von Funny Technology lief auch locker 24h oder länger. Profil war schneller. Bilder etc sind natürlich ein Problem
ReplyDeleteI think without Tor has been actually faster for me, though it's not speedy because, well, Google. I doubt that my internet speed is a limiting factor, so maybe this will be useful for others.
ReplyDeleteOver 35K posts in 14 hours without Tor with 1.8.4, and it's doing more work, finding more images (almost 2700 more so far) than 1.8.3 did. I have around 45K more posts to download (based on the 1.8.3 refresh numbers) before I can start downloading new images and videos. At my current rate (looks like around 2500/hour) I have about 18 hours left before I can start downloading more images and videos.
The image and video download has been much faster, and no new videos were discovered in the first 35K, so it's mostly images. Therefore downloading the newly-discovered images should be relatively quick. (Famous last words, I know.)
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/Ay8twiZceCrMBWxeiBIy1qTKEcHVLWW9Xq-0qHObgT15aAFkNhl4ds-FweQw1fTvoma_8FAkqthWxfraFqV0f5sSFSIgPFQD6xO_=s0
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/RsXUWfrbNycAMwzrzimPJ9lfYDsqlWHEghj90ciwkMiiZLIQ3eE8C5_YY75spwiWmdqaRXBuYYPEFKjuPRx2I4-EWSmqXeIO-qja=s0
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/I4l8PMY3hdbECvowntDoxf-_nbzfI2CmTK7y8g99fLkkKs0_zDotvAhXGN2SObDz7n_lzsD1Tb0GQUSQn6YcU0bWsIWr1TPzOv-n=s0
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/UpD8aLDou_0BpTBD9Ldum8VfVSia6roUewhTa5XvdfYWTQ_BhY5RicQOnuMMM8WG6cIvgdQT5srKylIHAiWZ-mFq3mfcfzcz80Zt=s0
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/Vq3YG3O9ccyTeVwfOBZ4ej09DOgW2ZXu_3OWon6HY-7wJuB2OFJgPj4scngNcfO-Hi6-UDlG4ubKW3WWT_JeHF4mSmDHdA6c5GQT=s0
Oh my …
ReplyDelete45,000 posts and counting.
Still not finished after approx. 40 hours.
What if I quit and update to 1.8.4 now? will it start all over again?
https://lh3.googleusercontent.com/fhcARpw8pD4UT-k3qQISQBsVijSauCA9bA4JBTHxD6OPeq8btvM_KuctM4SuN_DuiqcU43ko3KSsFT822yq5OZm8iPOeK6z1q_VL=s0
Gerhard Torges yes, it will start again.
ReplyDeleteThat's bad, Alois Bělaška.
ReplyDeleteI wonder why there are so many posts.
45,000 in 8 years, that's 15 posts every day.
Oh, and while you're here:
While counting it hasn't downloaded any content yet, right?
Gerhard Torges not sure.
ReplyDeleteNot sure, Alois Bělaška?
ReplyDeleteYou made that thing! 😁
Gerhard Torges the problem is not everybody is served the same Google+ web application. Impossible to fine tune the app for all G+ versions.
ReplyDelete"While counting" it is downloading all the content except images and videos. That is a count of downloaded data. As it goes, it records images and videos to download afterwards, which is much faster.
ReplyDeleteThanks for the clarification, Alois Bělaška and Michael K Johnson.
ReplyDeleteI think the best idea is to let it finish this run, then upgrade to 1.8.4 and try another one.
Gerhard Torges that is what I would do in your place. Make sure to download images and videos before running again with 1.8.4
ReplyDeleteWill do, Michael K Johnson.
ReplyDeleteI still find it kind of strange that downloading the "text part" is so much slower than downloading images and videos.
Gerhard Torges for posts we have to slowly communicate with G+ servers, images and videos are downloaded from more powerful Google Image proxy servers.
ReplyDeleteDo you think things are running slowly because many people are using the Exporter right now?
ReplyDeleteGerhard Torges my job is building cloud software that handles billions of events per day. This difference doesn't surprise me in the least. The images and videos are static stream files that have mostly already been copied around the world on content delivery networks designed to get lots of unchanging bits to you as fast as possible. The posts are from querying databases for lots of little objects that can change at any time and so have limited ability to be cached, and then assembling them into data in the format the part of the program that runs in your web browser (or on your phone) wants. That's not only slower, but also was designed to run at human usage scale, not as a lazy bug, but taking advantage of that characteristic to bring other benefits.
ReplyDelete49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/CKjvgWpHG2e6V6QEBgnVN_pHVNPEOjbvcvHs6YvHlkqyGoksQRpU-PGTILWx8Ze1ojPHGBh4luse-ml6uUzyi2Jr5DOaK6c1bQHX=s0
49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/6p1CF-HGO4gOKg_On07hxPFRydLMQ0jdQaUReq_0VFN58tVgIA8SR5mO-ljYp9OzUuCKlkT-nyHHTFxu6wQrv_c1u_Gn9bGbGO9U=s0
49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/RCWr90U-xghntdnLjoDh_pBgyVgVo3v-WEQm0aMtClHtNycCYcTgc0fm7y4fe414NciFpoJllRVTjEPJ_hKarn0qeCK3aVW0SsKG=s0
49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/zmbCNfSz5iB5OmMp_kv0AzdHQnIzRqpO5DJTMo4tC1ZJF_cG0YPUZxAwkgBVTfunirV6LOLFVxlm4pXDQEu_U4X6NZE9LCDV3ana=s0
49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/x3849RLX3uuM41q_MxEihE9gsr8swhQ8qOOTYtIpoVuUcKtYEVDjvghZ4bygyfEGyARR-K-mbOJfDvg6_6KsfIWeSYqr_it38xp3=s0
49,000 and counting.
ReplyDeleteAre these really all my posts?
Unfortunately, Google doesn't provide means to see how many posts a user created.
https://lh3.googleusercontent.com/ihjB46D-JB3s3nV0zUgij9_PM5y13H7WyN4FSAuT6n7aQClAil_4R6N9ocXBw9cPzmKPEh03cbbTjVIcjBV2sEnPM_KFtdIPuzFz=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/_r8RI0O4evfczNPqcPhsZt45ZJB11crXbmiNf_2LPitd-ap0-Q8wNowNGFgwDGb3hIRfMHwKEvvrbQsAemTJ-yeH60kc7Jll4Jq6=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/9hDDyiljjAF5NxkbaqcIShF648FW3oywNM3vqATNi9lOFl6_gojSMsTw1AMKo5UDMXHtes3Qmj16Z4hO31cRikOiOyj5_Eq4ztwN=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/mc999kCIQfNX1AkuTNLlXEKqvvkHPipAI2goukPWJKeiPj1v95EU3P8t2vwt2c7PAxnhI88S1Sj-XLuclmToXVqlVnnFFVWd6jD0=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/_3_TFZV9qRd-EYLwtj-GVSLGt3qYpau8P40SDu83ymypXgpxaLUU9iJmVjFz13cX-hQ3ue0M78syC_ASiwpANtbuBeIKT4lIw29U=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/kAbrD5gwjjZ9kuhkosmbsrKqfL-IAzQtAQ7AG57li_fsMFrQl_acO6AlqO2DJDJ78iSRDwMEjn0kMcMe4Hud4QvoMzmxdPbP9oIe=s0
Looks right to me. The timestamp of the database files never gets older than 1-2 minutes.
ReplyDeletehttps://lh3.googleusercontent.com/nwZk0W5lX7a3chddFtTjDMYr84hcq2tIlV6AiJgkFnPv-xIIuoTJ_pkJb01Vzdf_1PV0HQnzrANCiObT7WohUpqPWz_atJ9SqMWK=s0
Gerhard Torges if you do a takeout, you'll see what Google finds for your posts. I would agree that 49,000 seems like a lot. I rescued a bunch of maker communities and between those communities I have imported into their new home just over 50,000 so far...
ReplyDeleteI would strongly suggest that besides using the exporter, do a JSON-formatted and an HTML-formatted takeout. HTML should give you something you can unpack and browse on your computer, and JSON is more likely to work with tools people develop later.
"Belt and suspenders"
I did JSON takeouts on March 14 and 30, Michael K Johnson.
ReplyDeleteGoogle said both times there are errors in the G+ stream part.
Nonetheless I'd like to check the number of posts.
Do you have any clue where I should look?
Gerhard Torges I glanced through your profile. At least recently, you reshare as aggressively as some people retweet. All those reshares are included in your posts. If you have been as active historically as recently, that number isn't obviously wrong.
ReplyDeleteWell, that makes sense.
ReplyDeleteStill running …
ReplyDeletehttps://lh3.googleusercontent.com/mgW0_1NqwHy9gWnOFcNvws10vZtPE0MFJBskAvzYXpBhtuDvIUIxlxdag15eHMXFUv7XJKVtiAGSyzmFFtxcpRo9qf4XPzk89vlS=s0
Still running …
ReplyDeletehttps://lh3.googleusercontent.com/IC9QpsBlvpa5zoGrsdsmLzcPaY40FQbWdX4WiQ1OPS4PJi8YnudVxSkurh5Vo13XkO89gECZHWHcdr58wDuCPUXpQeHbEwUrMMBD=s0
Still running …
ReplyDeletehttps://lh3.googleusercontent.com/_ZmEOKEbbxqU5n8XVjEWtKRFEVBpRSxzytP4P8l4QGMnn-sY6b4kSRclfI7R7lztMWFNRBQD2QtyT_0fsZg1P3rgbXmWP5iYPCd1=s0
Still running …
ReplyDeletehttps://lh3.googleusercontent.com/XVeLytD_9HINfknN1sYkPbBgplasap51DXhlrbaBOzzGEZUpPlY8d4YV7GrwG8575ezAaxnDKwGkEQxS94adYQCbNpua4OshIfhp=s0
Still running …
ReplyDeletehttps://lh3.googleusercontent.com/nPC3YIBFh9B5lIjUjOeH3q4TvFyE9rx9vo4l3Z0R8UEg0w22IPsC8-qsThnPrH45Lpdp9AKjj1i6TcXDlk185jr2ITsfuSrufOan=s0
Still running …
ReplyDeletehttps://lh3.googleusercontent.com/yo_CjkA1MOb8OyX7tzGlYW6-C1UemSAQA-3P4EkwPtNgBFTxb9PG3q26LNvjBZzPSAWdlESmVRwbdNI8wADt8fX1cnA6-AjOkEdN=s0
I'm getting a bit nervous.
ReplyDeleteHours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/EI1Pufzyvjjnebol5L5ldMlbxEj-bZWUhSJ63gAoDtsYp7ULQRd8enFmmxAf8qWekigKgFjYg20BAKPnR5VAyYplzU4aRUDKK3R5=s0
Hours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/CYwosBrqjuDrkW4tAx-aR1jF4aHyCM2mIPQHHvmK5Tg9z3jbUSbaaEHHtCVb3AgPYXsnszs_z0q1DsPNxgCCADzXCphWALWkMoRD=s0
Hours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/rU2SbU9GyQ9FmqJWLOkpiPinbxlB7ZhdL3GwgQ5wngfCMfn2yAVjX9r720t1FrenFUGABLGyOVhJBp1n_oK6JDQ3QF1Ooz6kq51j=s0
Hours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/OLKTIWQ--KoCEwIBrDQz7C27Qr3JVocINJ2nt4bztiFJlKRC45d3AABQsh684KPGvRvSdx5idtOLG0RnRV2H9x4Qx9qwieg1EXPx=s0
Hours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/7w78rDcG7Yxbs79AxmE-tjo3L2Zo--Qc0kmpm3d3uin_qQERliwOERnUWl6-tY6f6-tTLtXEqcHuy8VWT5wHhfh70fgBM9f2YlA3=s0
Hours and hundreds of posts later.
ReplyDeleteThe database's size barely changed..
https://lh3.googleusercontent.com/V9kuSzGJrBj17wgQMR_6OBcFME3SDVOKUMxucLKnsgC6vPERAYvL8HDHfx6bQCa0g7fBqeTJWuE4qqKfzYC67S3eaSx-bJsXxdL1=s0
"At my current rate (looks like around 2500/hour) I have about 18 hours left before I can start downloading more images and videos."
ReplyDeleteIt has slowed down for me. I downloaded another 30,000 or so in about 16 hours, and I have about 15,000 left to download before I can download new images (6116 more found so far) and videos (2 more found so far).
When it comes to "belt and suspenders" github.com - FiXato/Plexodus-Tools has tools meant to work with the takeout data.
ReplyDeleteMaybe Google is already pulling off servers from the net.
ReplyDeleteGerhard Torges Please DO NOT spread unsubstantiated rumours through idle speculation.
ReplyDeleteIt wasn't meant that way, Edward Morbius.
ReplyDeleteMy Download counter is now at 62,000 posts for my main (this) profile and still counting.
This is extremely fishy.
Alois Bělaška does 1.8.5 have anything relevant here?
ReplyDeleteAdditionally, the database file is still about 69 MBytes in size.
ReplyDeletehttps://lh3.googleusercontent.com/sUowR6XL4j_vIDSkRM32-j_RAj9K5R8NlzsvmiKpg0gcP-0khU__Bfc_ZcyoIrokBTlqJYr384EB9edKp-wAAiOXrGfgGKyCmJ6_=s0
Additionally, the database file is still about 69 MBytes in size.
ReplyDeletehttps://lh3.googleusercontent.com/Ffvml16N71kXnxjf_-jI0w1CG6m92IknEv4bDt_qRy6SIX6reG4sM15SDeeoiVKKufPXXrB9ihdYN796K6zfxlQ_aC1VigTnLlyf=s0
Additionally, the database file is still about 69 MBytes in size.
ReplyDeletehttps://lh3.googleusercontent.com/PMG1CiVChyuZIDKcGSkY50fCOb3JsvoKWPTunZr2h8Nt2809sXiyDNEOBiuvBmq5nA-a0WaUckcVkp-5gIAEWC1EaV2ykIswrmqN=s0
Additionally, the database file is still about 69 MBytes in size.
ReplyDeletehttps://lh3.googleusercontent.com/TfwugkPqUgX0_nyehxNLD_duouyrHZVfsxmwz_mG56gUz8Jw0tCtWHAWIBCPmU5oa_KRRqQ4OK9bDXnM7vmwUBLPmpD2vpKsGDDK=s0
Additionally, the database file is still about 69 MBytes in size.
ReplyDeletehttps://lh3.googleusercontent.com/TzRavbrYtXrIiSPUg_mFUCsMisbM8XXcoRxZ4sa8AUtmuUmhxJPd5SWuYEUenlKvALUhDC69uhdHxMEr6ORVOJHLvFk0NK5D8kf-=s0
The data folder at 17:56 today:
ReplyDeletehttps://lh3.googleusercontent.com/C9NJRBYW8rhZPQ58aOIPj6SosHrKY9vzvyT5LNiGfNzryrTZsS9RHsbTA0w9UmZwxiIw5a-yjeTG5ZRPRMZIlNKswF8Rpr-ry-eX=s0
The data folder at 17:56 today:
ReplyDeletehttps://lh3.googleusercontent.com/Lpw0pOZ_WgC52DRt9HB2CF4xCC-6MxQgQfisG_4W-BZQ5x8MdXdY1K6CUJyJ4LGE3tOHvCZhCMTUfGID0vnoYueFZgvK7C4OHx_g=s0
The data folder at 17:56 today:
ReplyDeletehttps://lh3.googleusercontent.com/qW3ByZNMcVqjFovMg5yw-UoH-u0BW-qrWTrafcJzAq8J7dZthS-Obg6tYQtAVXkLQO6DeGGR-YwF5StLLBN3SevMq_CbXIzoIjnd=s0
The data folder at 17:56 today:
ReplyDeletehttps://lh3.googleusercontent.com/-bjNsHPE5N_bvkbUInNhhJh8eqpnF0rBwc-d3V8TB3MMWts6-w_JydN7Qm6-ysCYabyuUaw_X7gs-12785SoWZBiSnWFWdW2_Pa3=s0
The data folder at 17:56 today:
ReplyDeletehttps://lh3.googleusercontent.com/LCgyzIe8QE77EtS6gn0zx0rP7rUPKVLUXzQfJVLkg49zgz39iDRBqJqiVsPuQRfLDStAkEMKp-96h_vrDUGRdUizgi7PJSRvsUn7=s0
Same folder at 23:34:
ReplyDeletehttps://lh3.googleusercontent.com/nnDzymbGW2o6ftD677-4cezVCFskrbc5WsBLq4Tk8cy9p0br7IrLgyG91tTm06Y0kVtWSwmSwfkbkjXsGgHiQ_omMl9cVChsQ13V=s0
Same folder at 23:34:
ReplyDeletehttps://lh3.googleusercontent.com/mYDCCIN1khhkVdRWFb0ve_ETkdtgpi-eqc60S-gbu3o1jK6QHPQNnyJdw3n1JPpbIoK0O4suwDsPDbSTwftZx5we67GSq-m7hbDD=s0
Same folder at 23:34:
ReplyDeletehttps://lh3.googleusercontent.com/VWO4qZ1wvVwfLnG0OSXiFkj366zUWKEtcWohydOXpO4uXvJpAe6TINjIN8LmzsjlHMuL5qCbmZsNwvubdIJLkNhirqxv5UFuzrfw=s0
Same folder at 23:34:
ReplyDeletehttps://lh3.googleusercontent.com/YKSRkzmxymm_nEApV8Qdx2GnG6v4RrK_KeLXIONKcmDiDKOFi49WeYuPtU9mzK3EwX9dcuVOU2n9djvXv_6xqRbs5B1uIJNB_0ad=s0
Same folder at 23:34:
ReplyDeletehttps://lh3.googleusercontent.com/5QXE1FhIk_NW9dJR12V1wAY5VE5u5g61ufqNKRdW45kNYbdmTEKyY7Odd9b6QgMJ6WLb3-CNsvcBrmfOtIQZay3Kzacic9-qCV57=s0
1.8.5 may help. I’ve fixed a few bugs there but no guarantee.
ReplyDeleteI do hope so.
ReplyDeleteShould have started earlier.
Does it scan the profile from new to old posts, Alois Bělaška?
ReplyDeleteGerhard Torges it scans the feed as you see it on the web, from newest posts to the old ones.
ReplyDeleteBy parsing the HTML?
ReplyDeleteGerhard Torges by using G+ web application api.
ReplyDeleteI started over with 1.8.5.
ReplyDeleteJust at roughly 3,000 posts now. 😕