This article is a mirror article of machine translation, please click here to jump to the original article.

Architect_Programmer_Code Agriculture Network»Architect › Programming › Java › The crawler just written in Java, now only has the ability to download images (depth 1), will ...
View: 32261|Reply: 11

[JavaSE] The crawler just written in Java, now only has the ability to download images (depth 1), and will continue...

[Copy link]
Posted on 6/3/2015 2:38:12 AM | | | |
As the title suggests

crawler.rar (62.53 KB, Number of downloads: 5, Selling price: 2 Grain MB)




Previous:Do you know what web containers asp.net have? Except IIS
Next:deduplicate data and return the collection

Related Posts
Posted on 6/3/2015 9:05:36 PM |
Simple implementation that doesn't depend on other packages

  1. package test;

  3. import java.io.File;
  4. import java.io.IOException;
  5. import java.io.InputStreamReader;
  6. import java.net.MalformedURLException;
  7. import java.net.URL;
  8. import java.util.regex.Matcher;
  9. import java.util.regex.Pattern;

  11. import javax.imageio.ImageIO;

  13. public class Test {
  14.         public static void main(String[] args) {
  15.                 String web="http://www.itsvse.com/";
  16.                 try {
  17.                         URL url=new URL(web);
  18.                         InputStreamReader reader=new InputStreamReader(url.openStream());
  19.                        
  20.                         StringBuilder builder=new StringBuilder();
  21.                         char[] buff=new char[1024];
  22.                         int n;
  23.                         while((n=reader.read(buff))!=-1){
  24.                                 builder.append(buff,0,n);
  25.                         }
  26.                        
  27.                         Pattern pattern=Pattern.compile("<img.*?src="(.*?)(gif|png|jpg)"");
  28.                        
  29.                         Matcher m=pattern.matcher(builder);
  30.                         while (m.find()) {
  31.                                 String u=m.group(1)+m.group(2);
  32.                                 System.out.println("dowing.."+u);
  33.                                 URL img=null;
  34.                                 if(u.startsWith("http")){
  35.                                         img=new URL(u);
  36.                                 }else{
  37.                                         img=new URL(url,u);
  38.                                 }
  39.                                 ImageIO.write(ImageIO.read(img), m.group(2), new File("D:/img/"+System.currentTimeMillis()+"."+m.group(2)));
  40.                                
  41.                         }
  42.                        
  43.                 } catch (MalformedURLException e) {
  44.                         // TODO Auto-generated catch block
  45.                         e.printStackTrace();
  46.                 } catch (IOException e) {
  47.                         // TODO Auto-generated catch block
  48.                         e.printStackTrace();
  49.                 }
  50.         }
  51. }
Copy code
Posted on 6/4/2015 7:19:48 PM |
Delver_Si Posted on 2015-6-3 23:57
Your original code development efficiency is too low, bad review

I didn't want to say anything, but you said that development is inefficient。。。。。

The program requires the quality and performance of the code, and in the end, it has few functions, poor expansion ability, and poor performance


Run 10 times in a row, ignore network latency each time, ignore local saves, and only calculate the time to parse html documents, your program is far from it.
Also, there are errors in your code, so I won't say anything

Posted on 6/3/2015 1:00:52 PM |
Delver_Si Posted on 2015-6-3 12:57
How is the study?

I didn't install eclipse and looked at it in a notepad, first grabbed the HTML source code of the web page, then got the value after src, and then saved it   

I don't know if it's right  
Posted on 6/3/2015 7:49:23 AM |
Can images in PNG format be grabbed?
 Landlord| Posted on 6/3/2015 10:17:34 AM |
Small knife flying knife flying flying published on 2015-6-3 07:49
Can images in PNG format be grabbed?

Yes, I haven't judged the suffix now, all of them are saved as jpg, in fact, the png image can be opened with a jpg suffix, and I will improve the suffix
Posted on 6/3/2015 12:52:13 PM |
Let me study the research
 Landlord| Posted on 6/3/2015 12:57:13 PM |
Small slag Posted on 2015-6-3 12:52
Let me study the research

How is the study?
 Landlord| Posted on 6/3/2015 1:05:27 PM |
Xiao Zhazha Posted on 2015-6-3 13:00
I didn't install eclipse and looked at it in a notepad, first grab the html source code of the web page, then get the value after src, and then save the rough ...

That is true
Posted on 6/3/2015 9:12:09 PM |
microxdd posted on 2015-6-3 21:05
Simple implementation that doesn't depend on other packages

This is the rhythm that forces me to install myeclipse!
 Landlord| Posted on 6/3/2015 11:57:27 PM |
microxdd posted on 2015-6-3 21:05
Simple implementation that doesn't depend on other packages

Your original code development efficiency is too low, bad review
Disclaimer:
All software, programming materials or articles published by Code Farmer Network are only for learning and research purposes; The above content shall not be used for commercial or illegal purposes, otherwise, users shall bear all consequences. The information on this site comes from the Internet, and copyright disputes have nothing to do with this site. You must completely delete the above content from your computer within 24 hours of downloading. If you like the program, please support genuine software, purchase registration, and get better genuine services. If there is any infringement, please contact us by email.
Mail To:help@itsvse.com