What is a Web Crawler?

A Web Crawler is nothing but a software program which is also known as a Bot. A Web Crawler's task is to crawl from webpage to webpage by using the links available on the anchor tags of the webpage. Sometimes, it is also referred as Web Spider. Web Crawlers are mostly used by Search Engines to index the web pages.

Internet as a Graph.

An Internet is network of networks and Web Crawler considers Internet as a Graph (Data Structure). A Web Crawler traverses the Internet in the same way we traverse the Graph i.e by using the algorithms like Breath First Search or Depth First Search. Internet is similar to a graph where every web page is like a node in a graph and these nodes are connected to each other by edges. In every web page, we can find an HTML Anchor tag which links to some other page, either of same domain or of any other domain and these links works as edge between two web pages.

Now, we came to know that Internet is similar to Graph and hence their technique of traversal is same. But as in Graph we have to provide a starting point for traversal which is also similar in case of Web Crawler. Yes, we have to feed some urls(also known as Seeds) as an input to the crawler. By doing this, we give the crawler a start point to commence his work.

How it Works?

The concept used by the the Web Crawler is quite simple:

1. Select a URL from the set of urls.
2. Download the web page associated to that link.
3. Extract the links from the web page.
4. Filter the links by removing the links which were previously traversed.
5. Store the non-traversed link into the set of urls.
6. Fetch the next link from the set and repeat the same procedure from Point 1.

Yes, development of a Web Crawler is very easy, but it will be a simple Web Crawler. To build a Crawler which are used by Search Engines is a big challenge. As to maintain a Search Engine updated with latest web pages, a crawler has to traverse the pages in high scalability like crawling thousands of pages every second. If we try to do this then there is high possibility that Web server will crash. So one of the way is to distribute the downloads with multiple computers which will reduce the load.

Crawling Policies.

Every Web Crawler has to follow some policies. These policies helps crawler to work smoothly. Below are to four policies that a crawler has to follow.

• Selection Policy:
  A Crawler has to select which pages it needs to crawl. Every crawler has to obey the Robot Exclusion Protocol or robots.txt file
• Re-Visit Policy:
  A Crawler has to revisit the web pages to refresh the changed contents of the pages.
• Politeness Policy:
  A Crawler must not disrupt the performance of the website or server
• Parallelization Policy:
  A Crawler must execute multiple processes to enhance the download rate.

Existing Web Crawlers.

As mentioned earlier, a Web Crawler is mostly used by Search Engines. Following are the list of crawlers used by the most popular Search Engines:

• Google: Googlebot
• Yahoo: Slurp.
• Bing: BingBot.

Adding background image to JFrame - Java

This guide is made to add an image as a Background to JFrame.

So, to achive it, we will modify the ContentPane of the JFrame, by using the JFrame's setContentPane() method. We will also require an additional class which will be our ContentPane of out JFrame. This new class will extend JPanel class.
i.e Image -> JPanel -> ContentPane of JFrame = Background Image on JFrame

Here we go,

• In the constructor of the Main class, we will set the contentPane() and also pass the path of our image file as String.



 public Main() {
 
  setContentPane(new BgPanel("image.jpg"));  // setting new ContentPane
  setVisible(true);                          // display the JFrame
  setSize(500,300);                          // set Size of the JFrame
 }




• Now, in the constructer of out BgPanel class we will get the image from the path provided from the Main class. It can be done by using ImageIo class. It has a static method read(), it accepts a File object and  returns a BufferedImage as a result.

 
 img = ImageIO.read(new File(path));   



• IOException has to be handled for the read() method. IllegalArgumentException can also be thrown by read() method if the input is null.
  But our input won't be null as we are passing the image path as parameter. But IOException has to be handled.
  
  

   BgPanel(String path) {
   
    try {
    
     img = ImageIO.read(new File(path));
    } catch(IOException e) {
    
     e.printStackTrace();
    }
   }
  

• Now we need to paint the image on the JPanel, Therefore, we will be overriding the paintComponent(Graphics g) method. A call to the super class is made to clear the background of the component and the drawImage() method is used to draw an image on the JPanel.

   @Override
   protected void paintComponent(Graphics g) {
   
    super.paintComponent(g);
    if(img != null) {
    
     g.drawImage(img,0,0,this.getWidth(),this.getHeight(),this);
    }
   }

Here is the final code:

 /*
 * Author: Ashish Gundewad.  
 */
 import java.io.*;
 import java.awt.*;
 import javax.swing.*;
 import javax.imageio.*;

 public class Main extends JFrame {

  public Main() {
 
   setContentPane(new BgPanel("image.jpg"));
   setVisible(true);
   setSize(500,300);
  }
 
  public static void main(String [] args) {
 
   new Main();
  }
 }

 class BgPanel extends JPanel {

  private Image img;
  BgPanel(String path) {
 
   try {
  
    img = ImageIO.read(new File(path));
   } catch(IOException e) {
  
    e.printStackTrace();
   }
  }
 
  @Override
  protected void paintComponent(Graphics g) {
 
   super.paintComponent(g);
   if(img != null) {
  
    g.drawImage(img,0,0,this.getWidth(),this.getHeight(),this);
   }
  }
 }

Tic Tac Toe Game in Java

Hello everybody,

                   Here I want to share 'Tic Tac Toe' game also known as 'Noughts and Crosses' which I developed in Java (Swing).

/*  
 * Author: Ashish Gundewad.  
 * Program: Tic Tac Toe Game.  
 * Note:  Use Java JDK 1.6 and above for proper execution of the code.  
 */  
   
 import java.awt.*;  
 import javax.swing.*;  
 import java.awt.event.*;  
   
 public class TicTacToe extends JFrame implements ActionListener {  
   
      JButton b[]=new JButton[9]; // Array of 9 JButtons.  
      JPanel p;  
      JLabel l;  
      int fin=0;  
      int player,starter;  
      int []lock={0,0,0,0,0,0,0,0,0};  //Locks the button to overwrite the markings.  
      int btn=0;  
      int winX[]={0,0,0,0,0,0,0,0,0};  //Stores current marked location of 'X'  
      int winO[]={0,0,0,0,0,0,0,0,0};  //Stores current marked location of 'O'  
      int isWinX[]={0,0,0,0,0,0,0,0,0};   
      int isWinO[]={0,0,0,0,0,0,0,0,0};   
        
      // [][]winPattern is a matrix(3x3) which stores the all possible pattern to win the game.  
      int [][]winPattern={{1,1,1,0,0,0,0,0,0},{0,0,0,1,1,1,0,0,0},{0,0,0,0,0,0,1,1,1},{1,0,0,1,0,0,1,0,0},{0,1,0,0,1,0,0,1,0},{0,0,1,0,0,1,0,0,1},{1,0,0,0,1,0,0,0,1},{0,0,1,0,1,0,1,0,0}};  
        
      TicTacToe(int pl) {  
        
           starter=player=pl; // Starting player; 1 for 'X" and 0 for 'O'  
             
           // Create GUI  
           l=new JLabel("",JLabel.CENTER);  
           l.setBackground(new Color(215,192,195));  
             
           l.setFont(new Font("Arial", Font.BOLD,20));  
           p=new JPanel();  
           p.setLayout(new GridLayout(3,3));  
             
           if(pl==1)  
                l.setText("Player 1's Turn");  
           else  
                l.setText("Player 2's Turn");  
             
           setLayout(new BorderLayout());  
           setVisible(true);  
           setSize(500,500);  
           setTitle("Tic Tac Toe");  
             
           add(l,"North");  
           add(p,"Center");  
             
           for(int i=0;i<9;i++) {  
             
                b[i]=new JButton();  
                b[i].addActionListener(this);  
                b[i].setFont(new Font("Arial", Font.BOLD,80));  
                b[i].setBackground(new Color(215,192,195));  
                p.add(b[i]);  
           }  
             
           addWindowListener(new WindowAdapter()  
           {  
                public void windowClosing(WindowEvent we) {  
                  
                     System.exit(0);  
                }  
           });  
           // End of GUI code.  
             
      } //End of constructor.  
        
      public void actionPerformed(ActionEvent ae) {  
        
           for(int i=0;i<9;i++) {  
             
                if(ae.getSource()==b[i]) {  
                       
                     if(lock[i]==0) {   //Check wheather the button is locked.  
                            
                          lock[i]=1;    //If not locked, then lock it.  
                          if(player==1) {  //If current player is 'X'.  
                            
                               b[i].setText("X");  
                               player=0;  
                               l.setText("Player 2's Turn");  
                               getWin("X"); //check Winning pattern for 'X'.  
                          }  
                          else {            //If current player is 'O'.  
                       
                               b[i].setText("O");  
                               player=1;  
                               l.setText("Player 1's Turn");  
                               getWin("O"); //Check Winning pattern for 'O'.  
                          }  
                            
                            
                     }  
                }  
           }  
      }  
        
      //Method to check win pattern.  
      void getWin(String p) {  
             
           if(p=="X") {  
             
                for(int i=0;i<9;i++) {          //Get marked location of 'X'.  
                  
                     if(b[i].getText()==p)  
                          winX[i]=1;  
                }  
             
                for(int i=0;i<8;i++) {  
                  
                     for(int j=0;j<9;j++) {       
                            
                          isWinX[j]=winX[j] & winPattern[i][j];  //Perform 'AND' operation of current marked position with win Pattern.  
                                                                             //and store it in a array for later comparision.  
                     }  
                     int k;  
                     for(k=0;k<9;k++) {  
                       
                          if(isWinX[k]!=winPattern[i][k])               //Compare the result array with the win pattern.  
                               break;  
                     }  
                     if(k==9) {  
                       
                          System.out.println("X Won...");  
                          for(int j=0;j<9;j++) {  
                            
                               if(winPattern[i][j]==1) {               //Highlight the win pattern in green color.  
                                 
                                    b[j].setBackground(Color.GREEN);  
                               }  
                          }  
                          fin=1;  
                          int op=JOptionPane.showConfirmDialog(this, "X Won..!!! \nPlay Again?", "Tic Tac Toe", JOptionPane.YES_NO_OPTION,JOptionPane.QUESTION_MESSAGE);  
                          if(op!=0){  
                               System.exit(0);  
                          }  
                          else {  
                               dispose();  
                               new TicTacToe(1);  
                                 
                          }  
                          break;  
                            
                     }  
                }  
           }  
             
           if(p=="O") {       //Similar steps as above, but for player 'O'.  
             
                for(int i=0;i<9;i++) {  
                  
                     if(b[i].getText()==p)  
                          winO[i]=1;  
                }  
             
                for(int i=0;i<8;i++) {  
                  
                     for(int j=0;j<9;j++) {  
                            
                          isWinO[j]=winO[j] & winPattern[i][j];  
                     }  
                     int k;  
                     for(k=0;k<9;k++) {  
                       
                          if(isWinO[k]!=winPattern[i][k])  
                               break;  
                     }  
                     if(k==9) {  
                       
                          System.out.println("O Won...");  
                          for(int j=0;j<9;j++) {  
                            
                               if(winPattern[i][j]==1) {  
                                 
                                    b[j].setBackground(Color.GREEN);  
                               }  
                          }  
                          fin=1;  
                          int op=JOptionPane.showConfirmDialog(this, "O Won..!!! \nPlay Again?", "Tic Tac Toe", JOptionPane.YES_NO_OPTION,JOptionPane.QUESTION_MESSAGE);  
                          if(op!=0){  
                               System.exit(0);  
                          }  
                          else {  
                               dispose();  
                               new TicTacToe(0);  
                          }  
                          break;  
                     }  
                }  
           }  
           int k;  
           for(k=0;k<9;k++) {  
                  
                if(lock[k]!=1) {  
                     break;  
                }  
           }  
           if(k==9 && fin==0) {  
                System.out.println("Draw");  
                int op=JOptionPane.showConfirmDialog(this, "Match Draw..!!! \nPlay Again?", "Tic Tac Toe", JOptionPane.YES_NO_OPTION,JOptionPane.QUESTION_MESSAGE);  
                          if(op!=0){  
                               System.exit(0);  
                          }  
                          else {  
                               dispose();  
                               new TicTacToe(starter);  
                          }  
           }  
      } //End of winPattern method.  
        
      public static void main(String []asas) {     //Main method.  
        
           try {  
                //Applying Nimbus Look and Feel. requires Java JDK 1.6 and above.  
                UIManager.setLookAndFeel("com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel");  
           }  
           catch(Exception ex) { System.out.println(ex);}  
           new TicTacToe(1);     //Call to the constructor. Parameter 1 for Player 'X'.  
      }  
 }