How to Build a Search Engine Using Python

Introduction

A search engine is a software program that helps people locate information on the internet. In this tutorial, we will be building a python search engine. For this tutorial, we will be using the lycos.com search engine to scrape the search results.

Install modules

For this tutorial we will be using Flask as our primary web server, requests to make HTTP requests, and beautifulsoup as our scrapping library To install these modules run the commands below on your terminal.

pip3 install Flask 

pip3 install bs4

pip3 install requests

Create files

To begin first create a folder called flask-search-engine and inside it create a folder called templates and inside it create a file called home.html and results.html Now inside the flask-search-engine folder called app.py.

Site Inspection

Here we will go into the lycos.com site and inspect the page to get the specific tags and elements the site implements. Here is a screenshot of the site inspection depicting various site sections.

As you can see there is the main div called result-item and then inside we have the result-title, result-url and result-description those are the only elements we need for our project.

Create app

Now let’s first begin by importing a few modules in our app as shown below in the app.py file.

from flask import Flask, render_template, request 
import requests 
from bs4 import BeautifulSoup

Next, we need to initialize our application as shown below.

app = Flask(__name__)

Next is the main view that will crawl the site and scrape the required classes and elements and then render the scraped data to our template.

@app.route('/results',methods=['GET','POST'])
def results():
    if request.method == "POST":
        query = request.form['query']
        results = []
        page = requests.get('https://search17.lycos.com/web/?q='+query).text
        soup = BeautifulSoup(page,'lxml')
        listings = soup.find_all(class_="result-item")
        for content in listings:
            title = content.find(class_='result-title').text
            description = content.find(class_='result-description').text
            link = content.find(class_='result-link').text
            url = content.find(class_='result-url').text
            results.append((title,description,url))
        return render_template("results.html",results=results)

    return render_template("results.html")


if __name__ == "__main__":
    app.run(debug=True)

Create templates

Now let’s create a few templates, inside the home.html file we created insert the code below, which will be our home page for the search engine.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <style>
        div {
            text-align: center;
        }

        .search-bar {
            margin-top: 7%;
        }

        .search-input {
            width: 400px;
            height: 40px;
            font-size: 16px;
            padding: 5px;
            border-radius: 8px;
            border: none;
        }

        .search-button {
            background-color: #FF6F00;
            color: #FFF;
            font-size: 16px;
            padding: 10px 20px;
            border: none;
            border-radius: 8px;
            cursor: pointer;
        }
    </style>
</head>
<body>
    <div class="home">
        <h1>Bytexplain Search Engine</h1>
        <img src="../static/ai.png" alt="Bytexplain Search" width="150" height="150">
        <div class="search-bar">
            <form method="POST" action="results">
                <input class="search-input" type="text" placeholder="What are you searching for?" name="query">
                <button type="submit" class="search-button">Search</button>
            </form>
        </div>
    </div>
</body>
</html>

Next let’s create our results template, in this template we will have a search form and then we will render our results below it. Insert the code below in the results.html file we created earlier.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <style>
        body {
            color: #FFF;
        }

        .search-bar {
            margin-top: 1%;
            text-align: center;
        }

        .search-bar input[type="text"] {
            width: 400px;
            height: 40px;
            font-size: 16px;
            padding: 5px;
            border-radius: 8px;
            border: solid;
        }

        .search-results {
            margin-top: 50px;
            text-align: left;
        }

        .search-results h4 {
            font-size: 20px;
            font-weight: bold;
            margin-bottom: 5px;
        }

        .search-results h5 {
            font-size: 16px;
            margin-bottom: 5px;
            color: #70757A;
        }

        .search-results a {
            text-decoration: none;
            font-size: 14px;
        }

        .search-results a:hover {
            text-decoration: underline;
        }
    </style>
</head>
<body>
    <div class="search-bar">
        <form method="POST">
           
            <input type="text" placeholder="What are you searching for?" name="query">
        </form>
    </div>
    <div class="search-results">
        {% if results %}
            {% for result in results %}
                <h4><a href="https://{{ result.2 }}">{{ result.0 }}</a></h4>
                <h5>{{ result.1 }}</h5>
                <a href="https://{{ result.2 }}">{{ result.2 }}</a>
                <br/><br/>
            {% endfor %}
        {% else %}
            <h4>No results found.</h4>
        {% endif %}
    </div>
</body>
</html>