Web Analytics Made Easy -
StatCounter Multiple External Requests Downloading & Localizing Image - CodingForum

Announcement

Collapse
No announcement yet.

Multiple External Requests Downloading & Localizing Image

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Multiple External Requests Downloading & Localizing Image

    Hiya Everyone,

    I'm making a new help/tutorial/fan based website for an online based MMORPG I have an interest in. This game currently has 827 items in-game and I wish to localize them on my own server.

    http://domain.com/images/items/NUM/large.png

    The NUM field above is anything from 1-827. They also don't have the best servers limiting their new user API to a maximum of 100 per minute therefore I'd want to do somewhat the same and not 827 in less than a second.

    Is there any quick and simple hacks for this please?

    Best Regards,
    Tim
    Last edited by MrTIMarshall; Sep 20, 2016, 11:02 PM.

  • #2
    Loop 0 - 826 (+1) OR 1 - 827

    In that loop, sleep 1 second between requests. set_time_limit(0) (using apache - some webhosts may disable it though so use xamp etc locally) then use file_put_contents to save the data to file.

    Unless the items you're after are on a https connection then you should be ok to use file_get_contents() for the http download but it is not as configureable as cURL.
    "Tango says double quotes with a single ( ' ) quote in the middle"
    '$Name says single quotes with a double ( " ) quote in the middle'
    "Tango says double quotes ( \" ) must escape a double quote"
    '$Name single quotes ( \' ) must escape a single quote'

    Comment


    • #3
      Originally posted by tangoforce View Post
      Loop 0 - 826 (+1) OR 1 - 827

      In that loop, sleep 1 second between requests. set_time_limit(0) (using apache - some webhosts may disable it though so use xamp etc locally) then use file_put_contents to save the data to file.

      Unless the items you're after are on a https connection then you should be ok to use file_get_contents() for the http download but it is not as configureable as cURL.

      What you've mentioned I just did some research on how to achieve as my knowledge of PHP is extremely limited. Below is what I have so far from what you've suggested;

      PHP Code:
      <?php
          $x 
      1;
          for (
      $x 1$x <= 826$x++) {
            
      $url "http://domain.com/images/items/" $x "/large.png";
            
      // Get images and save
            // Somewhere
            
      sleep(1);
          }
      ?>
      They've got HTTPS and HTTP.

      Comment


      • #4
        If you want to "localize them" to your own server, why not download and host them instead of leeching off their hosting (which is what it sounds like you're talking about)?
        Walk the dark path, sleep with angels, call the past for help.
        https://cutcodedown.com
        https://medium.com/@deathshadow

        Comment


        • #5
          Originally posted by deathshadow View Post
          If you want to "localize them" to your own server, why not download and host them instead of leeching off their hosting (which is what it sounds like you're talking about)?
          To go through them all and download them manually would take quite some time.

          This site I'm working on is for the site I want to take the images from.

          Comment


          • #6
            Originally posted by MrTIMarshall View Post
            What you've mentioned I just did some research on how to achieve as my knowledge of PHP is extremely limited. Below is what I have so far from what you've suggested;

            PHP Code:
            <?php
                $x 
            1;
                for (
            $x 1$x <= 826$x++) {
                  
            $url "http://domain.com/images/items/" $x "/large.png";
                  
            // Get images and save
                  // Somewhere
                  
            sleep(1);
                }
            ?>
            They've got HTTPS and HTTP.
            I did say 0-826 OR 1-827. You went with 1-826?

            Anyhow... completely untested...

            PHP Code:
            <?php
            set_time_limit
            (0);  //Make sure this actually works - a simple script with sleep(62) would do the job
                
            $x 1;
                for (
            $x 1$x 828$x++) {
                  
            $url "http://domain.com/images/items/" $x "/large.png";

                  
            $Image file_get_contents($url);

                  
            // Somewhere
                  
            file_put_contents("/path/to/your/storage/$x/large.png");
                 print 
            "Saved $x <br>\n";
                  
            sleep(1);    //Will take several minutes to finish - but you can check the folder to see if new images are appearing
                
            }
            ?>
            To test the time limit function this would do the job:

            PHP Code:
            set_time_limit(0);
            sleep(62);          //Some servers have a 30 second limit, others have 60
            print 'It works';    //Or you'll get an error 
            "Tango says double quotes with a single ( ' ) quote in the middle"
            '$Name says single quotes with a double ( " ) quote in the middle'
            "Tango says double quotes ( \" ) must escape a double quote"
            '$Name single quotes ( \' ) must escape a single quote'

            Comment


            • #7
              Just curious.

              If you have to loop for x number of times, and sleep for 1 minute during the loop, wouldn't the browser complain of timing out, as it would take a long time for the server to respond?

              Comment


              • #8
                It could do yes... but that seems to vary between browsers and causes of timeouts. I couldn't tell you the difference in the way the browser reacts to a connection timeout or a response timeout.

                Either way with the code supplied, it will leave physical files on the disk of the target machine so the code can be adjusted to restart the procedure.

                But that one minute sleep provided above isn't part of the loop - it's a seperate piece of code just to test if the server will handle long script executions.
                Last edited by tangoforce; Sep 22, 2016, 09:14 AM.
                "Tango says double quotes with a single ( ' ) quote in the middle"
                '$Name says single quotes with a double ( " ) quote in the middle'
                "Tango says double quotes ( \" ) must escape a double quote"
                '$Name single quotes ( \' ) must escape a single quote'

                Comment


                • #9
                  Ah, I think @tangoForce's code explained the part I wasn't understanding... you're trying to make a program to DOWNLOAD the images, copying them to your server. You didn't actually come right out and say that so it was'nae all that clear what you were trying to accomplish.

                  You could use a meta refresh to do them in batches. I'd Mr. Scott things so you do half what the limit is and wait 50% longer for good measure; but since this is only something you'd do once there's no reason to fret that. 10 minutes or 30, what's it matter?

                  Something like (untested)

                  copy.php:
                  Code:
                  <!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
                  <?php
                  
                  define('MAX_COUNT', 827);
                  define('WAIT_BETWEEN', 90);
                  define('GET_PREFIX', 'http://domain.com/images/items/');
                  define('PUT_PREFIX', 'path/to/destination/');
                  define('FILE_SUFFIX', '/large.png');
                  
                  $count = isset($_GET['start'])) ? $_GET['start'] : 0;
                  $end = min($count + 50, MAX_COUNT);
                  
                  if ($end < MAX_COUNT) echo '
                  <meta
                  	http-equiv="refresh"
                  	content="', WAIT_BETWEEN, '; url=copy.php?start=', $end, '"
                  >';
                  
                  $title = Copy images ' . $count + 1 . ' to ' . $end . ' of ' . MAX_COUNT;
                  
                  echo ' 
                  <title>
                  	', $title, '
                  </title>
                  </head><body>
                  
                  <h1>', $title, '</h1>';
                  	
                  while (++$count <= $end) file_put_contents(
                  	PUT_PREFIX . $count . FILE_SUFFIX,
                  	file_get_contents(GET_PREFIX . $count . FILE_SUFFIX)
                  );
                  
                  echo 'Batch complete, ', (
                  	$end < MAX_COUNT ?
                  	'will do next batch in ' . WAIT_BETWEEN . ' seconds' :
                  	'All transfers complete'
                  ), '
                  </body></html>';
                  Does 50, waits 90 seconds, does 50 more, until done. I put all the values you might change or have need of changing up top in define for ease of access. I didn't do typecasting or validation of the $_GET value since I would assume you wouldn't have this public facing. A more robust approach would do that, and trap if file_get_contents fails... but again since this does'nae feel like something you'd do more than once in a blue moon, it's likely that doesn't matter. (in fact, I'd delete copy.php once you have the files)

                  Oh, and the limit per minute might not be bad hosting, but intentional to block DDOS and content theft... sadly what you are trying to do MAY be considered the latter unless you have explicit permission to do so.

                  -- edit -- an even better approach might be to loop until you have a "get" that fails, and then do the wait and resume. That's how I'd go about it if I were writing it as a shell script using htget.
                  Last edited by deathshadow; Sep 22, 2016, 09:58 AM.
                  Walk the dark path, sleep with angels, call the past for help.
                  https://cutcodedown.com
                  https://medium.com/@deathshadow

                  Comment


                  • #10
                    Originally posted by deathshadow View Post
                    Ah, I think @tangoForce's code explained the part I wasn't understanding... you're trying to make a program to DOWNLOAD the images, copying them to your server. You didn't actually come right out and say that

                    To be honest, given that this is the title of the topic:



                    and then Tim says:

                    Originally posted by MrTIMarshall View Post
                    Hiya Everyone,

                    I'm making a new help/tutorial/fan based website for an online based MMORPG I have an interest in. This game currently has 827 items in-game and I wish to localize them on my own server.
                    I'm not really quite sure how you can accuse Tim of not coming right out and saying what he means. Instead you've misunderstood for whatever reason. That doesn't matter (I ignore topics I can't get my head around so you're not alone) but turning it round and trying to pin blame on the person asking for help is not nice - especially with words like yours that I framed in my signature.
                    "Tango says double quotes with a single ( ' ) quote in the middle"
                    '$Name says single quotes with a double ( " ) quote in the middle'
                    "Tango says double quotes ( \" ) must escape a double quote"
                    '$Name single quotes ( \' ) must escape a single quote'

                    Comment


                    • #11
                      Originally posted by tangoforce View Post
                      I'm not really quite sure how you can accuse Tim of not coming right out and saying what he means.
                      The wording was either too convoluted or too broken for me to comprehend -- I've reread it, and you can see in my first question I didn't understand why he wasn't just downloading them; automated or no. What he was asking for was what I was asking why he wasn't doing it... it wasn't until your post that I realized he was trying to download and SAVE them locally.

                      "Localize" can be surprisingly vague, more so when there's incomplete snippets or no code to show what in fact is trying to be attempted. I was stuck with "He's grabbing them, so now what?!?" since nowhere does it actually say WHAT.

                      Honestly, methinks you just have a better crystal ball than I. I would never have divined that intent from the earlier posts in this thread.
                      Walk the dark path, sleep with angels, call the past for help.
                      https://cutcodedown.com
                      https://medium.com/@deathshadow

                      Comment


                      • #12
                        Originally posted by deathshadow View Post
                        The wording was either too convoluted or too broken for me to comprehend
                        FINALLY!!!! Now you know what I was trying to get across to you in another thread and how important CLEAR communication is

                        Originally posted by deathshadow View Post
                        Honestly, methinks you just have a better crystal ball than I.
                        Runs in the family... I call it an advanced level or threat perception and forevision.. others call it paranoia
                        "Tango says double quotes with a single ( ' ) quote in the middle"
                        '$Name says single quotes with a double ( " ) quote in the middle'
                        "Tango says double quotes ( \" ) must escape a double quote"
                        '$Name single quotes ( \' ) must escape a single quote'

                        Comment


                        • #13
                          Originally posted by tangoforce View Post
                          others call it paranoia
                          In programming that's not just a good thing, I consider it a prerequisite. ESPECIALLY with web technologies. There are so many scams and so many dirtbags out there, paranoia has to be standard operating procedure!

                          -- edit -- oh, and in the service, they call it "Situational awareness". Mine seems to have become dulled of late; more and more I find myself questioning if I'm even reading the same language as everyone else; I read the same text as others and come away with wildly different conclusions and interpretations.
                          Last edited by deathshadow; Sep 24, 2016, 03:21 AM.
                          Walk the dark path, sleep with angels, call the past for help.
                          https://cutcodedown.com
                          https://medium.com/@deathshadow

                          Comment


                          • #14
                            Originally posted by deathshadow View Post
                            oh, and in the service
                            AH HA!!!

                            There it is. That explains your bad tempered attitude towards normal people. I've known a lot of people from the armed forces over the years and they're all the same - can't relax, chill or enjoy life like normal people can and do - just like ex police officers. I just thought you were a snotty coder who liked to look down on people but now I realise that it's something that has been instilled in you over the years - which isn't your fault. You need to relearn how to integrate with normal people because the way you treat people isn't normal - it's militant and that works fine IN THE MILITARY but not in the normal civilised world.

                            Not all coders are the enemy!
                            "Tango says double quotes with a single ( ' ) quote in the middle"
                            '$Name says single quotes with a double ( " ) quote in the middle'
                            "Tango says double quotes ( \" ) must escape a double quote"
                            '$Name single quotes ( \' ) must escape a single quote'

                            Comment

                            Working...
                            X