Regarding grabbing images from Google (such those linked in the G+Exporter files)
Are there known tools around that can mass download images from URLs without triggering any kind of throttling or denial from Google's servers?
Are there known tools around that can mass download images from URLs without triggering any kind of throttling or denial from Google's servers?
I use JDownloader. Works surprisingly good for this purpose.
ReplyDeleteMartina Neumayer My Chrome is not happy about the .rar file for JDownloader and says "Virus detected".
ReplyDeleteLars Fosdal Well known false positive on windoze machines. I use Arch so I don't care about some viri-alerts ;)
ReplyDeleteHere is an official installer source:
jdownloader.org - JDownloader.org - Official Homepage
Martina Neumayer That's the page I used. It redirects to a Mega.nz download - which also warns about a virus.
ReplyDeleteLars Fosdal I don't have any warnings there. Just already tested now and nothing at all.
ReplyDeletehttps://lh3.googleusercontent.com/se-zswIpcTSYaoDXldTwrlbjn9nN4FP5h8oe8AVAKqLqR4dPuy6GizxiSrU1v1-8JcRPNuRBZQGV4mY=s0
Martina Neumayer
ReplyDeleteChrome Version 72.0.3626.14 (Official Build) dev (64-bit)
https://lh3.googleusercontent.com/KaxDDg6avqj6-0fx4gy6Jx0DzvI-vU4h8qwGh9F6ZxrbMuW-wjie8qk1Q8q5SMtqb5CWBRvs4jLHa2Q=s0
Lars Fosdal Yeap..the charms of having a windows system onboard ;)
ReplyDeleteBtw.. As the support stated on their website:
"..The installers on the main page asks you if you want to install a bundled software - for example a toolbar. This is the way we earn money. The installer shows you a screen, and you can uncheck the toolbar installation. So you you read the installer screens, you will not get anything unwanted installed.
We do not like this Adware-Driven model alot, but there is no real alternative right now. We will offer a kind of "Paid-JD-Premium" Version of JDownloader that has no ads, and a few extra features. But this Premium services will come when we finally release JD2 (it is beta right now)
Unfortunately, JD2 Development is slower than we thought, and so we offer Adware-free installers here until we finally prepare for JD-Premium.".
That's why this file can trigger a warning. Just uncheck this additional option and you're good to go.
One more thing.. You can use also this "jar" binary file. Just rpm on it, choose "open with.." and then select to open it with "Java(TM) Platform SE binary". Should works without problems.
The likelyhood of me installing something that has been flagged - correctly or incorrectly = zero. Call me paranoid, but 35 years of computing has made me quite careful.
ReplyDeleteLars Fosdal Careful and youre using windoze? hahaha good one.. You made my day xD
ReplyDeleteMartina Neumayer Ridiculing an OS = so mature.
ReplyDeleteLars Fosdal Nope.. not os. And mature is reading a text with understanding. ;)
ReplyDeleteLars Fosdal My occasional G+ web scraping of 3k - 50k URLs at 1s intervals has never triggerred throttling AFAIA.
ReplyDeleteWe might be helping someone meet their KPIs!
Edward Morbius Thank you - that sounds promising.
ReplyDeleteLars Fosdal Yeah. I've tried to avoid tempting fate by generally including a 'sleep 1' or 'sleep 2' term in my shell loops. But I've never seen limiting occur.
ReplyDeleteI think Google may impose a rate limit on their side via QoS throttling or similar mechanisms to about 1 query/s, based on behaviour whilst I've tested such scripts on small (10-100 query) runs.