Category: Uncategorized

Posted on

Python BeautifulSoup Examples

Python BeautifulSoup Examples

https://ift.tt/2UWwaPZ

Introduction

In this tutorial, we will explore numerous examples of using the BeautifulSoup library in Python. For a better understanding let us follow a few guidelines/steps that will help us to simplify things and produce an efficient code. Please have a look at the framework/steps that we are going to follow in all the examples mentioned below:

  1. Inspect the HTML and CSS code behind the website/webpage.
  2. Import the necessary libraries.
  3. Create a User Agent (Optional).
  4. Send get() request and fetch the webpage contents.
  5. Check the Status Code after receiving the response.
  6. Create a Beautiful Soup Object and define the parser.
  7. Implement your logic.

❖ Disclaimer: This article considers that you have gone through the basic concepts of web scraping. The sole purpose of this article is to list and demonstrate examples of web scraping. The examples mentioned have been created only for educational purposes. In case you want to learn the basic concepts before diving into the examples, please follow the tutorial at this link.

Without further delay let us dive into the examples. Let the games begin!

Example 1: Scraping An Example Webpage

Let’s begin with a simple example where we are going to extract data from a given table in a webpage. The webpage from which we are going to extract the data has been mentioned below:

Link to webpage: https://shubhamsayon.github.io/python/demo_html.html

The code to scrape the data from the table in the above webpage has been given below. 

# 1. Import the necessary LIBRARIES
import requests
from bs4 import BeautifulSoup

# 2. Create a User Agent (Optional)
headers = {"User-Agent": "Mozilla/5.0 (Linux; U; Android 4.2.2; he-il; NEO-X5-116A Build/JDQ39) AppleWebKit/534.30 ("
                         "KHTML, like Gecko) Version/4.0 Safari/534.30"}

# 3. Send get() Request and fetch the webpage contents
response = requests.get("https://shubhamsayon.github.io/python/demo_html.html", headers=headers)
webpage = response.content

# 4. Check Status Code (Optional)
# print(response.status_code)

# 5. Create a Beautiful Soup Object
soup = BeautifulSoup(webpage, "html.parser")

# 6. Implement the Logic.
for tr in soup.find_all('tr'):
    topic = "TOPIC: "
    url = "URL: "
    values = [data for data in tr.find_all('td')]
    for value in values:
        print(topic, value.text)
        topic = url
    print()

Output:

TOPIC:  __str__ vs __repr__ In Python
URL:  https://blog.finxter.com/python-__str__-vs-__repr__/

TOPIC:  How to Read a File Line-By-Line and Store Into a List?
URL:  https://blog.finxter.com/how-to-read-a-file-line-by-line-and-store-into-a-list/

TOPIC:  How To Convert a String To a List In Python?
URL:  https://blog.finxter.com/how-to-convert-a-string-to-a-list-in-python/

TOPIC:  How To Iterate Through Two Lists In Parallel?
URL:  https://blog.finxter.com/how-to-iterate-through-two-lists-in-parallel/

TOPIC:  Python Scoping Rules – A Simple Illustrated Guide
URL:  https://blog.finxter.com/python-scoping-rules-a-simple-illustrated-guide/

TOPIC:  Flatten A List Of Lists In Python
URL:  https://blog.finxter.com/flatten-a-list-of-lists-in-python/

✨ Video Walkthrough of The Above Code:

Example 2: Scraping Data From The Finxter Leaderboard

This example shows how we can easily scrape data from the Finxter dashboard which lists the elos/points. The image given below depicts the data that we are going to extract from https://app.finxter.com.

The code to scrape the data from the table in the above webpage has been given below.

# import the required libraries
import requests
from bs4 import BeautifulSoup

# create User-Agent (optional)
headers = {"User-Agent": "Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/31.0.1650.0 Safari/537.36"}

# get() Request
response = requests.get("https://app.finxter.com/learn/computer/science/", headers=headers)

# Store the webpage contents
webpage = response.content

# Check Status Code (Optional)
print(response.status_code)

# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")
# The logic
for table in soup.find_all('table',class_='w3-table-all',limit=1):
    for tr in table.find_all('tr'):
        name = "USERNAME: "
        elo = "ELO: "
        rank = "RANK: "
        for td in tr.find_all('td'):
            print(name,td.text.strip())
            name = elo
            elo = rank
        print()

Output: Please download the file given below to view the extracted data as a result of executing the above code.

Click to Download and View Output

Example 3: Scraping The Free Python Job Board

Data scraping can prove to be extremely handy while automating searches on Job websites. The example given below is a complete walkthrough of how you can scrape data from job websites. The image given below depicts the website whose data we shall be scraping.

Link to website: http://pythonjobs.github.io/

In the code given below, we will try and extract the job title, location, and company name for each job that has been listed. Please feel free to run the code on your system and visualize the output. 

import requests
from bs4 import BeautifulSoup

# create User-Agent (optional)
headers = {"User-Agent": "Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/31.0.1650.0 Safari/537.36"}

# get() Request
response = requests.get("http://pythonjobs.github.io/", headers=headers)

# Store the webpage contents
webpage = response.content

# Check Status Code (Optional)
# print(response.status_code)

# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")

# The logic
for job in soup.find_all('section', class_='job_list'):
    title = [a for a in job.find_all('h1')]
    for n, tag in enumerate(job.find_all('div', class_='job')):
        company_element = [x for x in tag.find_all('span', class_='info')]
        print("Job Title: ", title[n].text.strip())
        print("Location: ", company_element[0].text.strip())
        print("Company: ", company_element[3].text.strip())
        print()

Output:

Job Title: Software Engineer (Data Operations)
Location:  Sydney, Australia / Remote
Company:  Autumn Compass

Job Title: Developer / Engineer
Location:  Maryland / DC Metro Area
Company:  National Institutes of Health contracting company.

Job Title: Senior Backend Developer (Python/Django)
Location:  Vienna, Austria
Company:  Bambus.io

✨ Video Walkthrough Of Above Code:

Example 4: Scraping Data From An Online Book Store

Web scraping has a large scale usage when it comes to extracting information about products from shopping websites. In this example, we shall see how we can extract data about books/products from alibris.com.

The image given below depicts the webpage from which we are going to scrape data.

Link to webpage: https://www.alibris.com/search/books/subject/Fiction

The code given below demonstrates how to extract:

  • The name of each Book,
  • The name of the Author,
  • The price of each book. 
# import the required libraries
import requests
from bs4 import BeautifulSoup

# create User-Agent (optional)
headers = {"User-Agent": "Mozilla/5.0 (Linux; U; Android 4.2.2; he-il; NEO-X5-116A Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30"}

# get() Request
response = requests.get(
    "https://www.alibris.com/search/books/subject/Fiction", headers=headers)

# Store the webpage contents
webpage = response.content

# Check Status Code (Optional)
# print(response.status_code)

# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")

# The logic
for parent in soup.find_all('ul',{'class':'primaryList'}):
    for n,tag in enumerate(parent.find_all('li')):
        title = [x for x in tag.find_all('p', class_='bookTitle')]
        author = [x for x in tag.find_all('p', class_='author')]
        price = [x for x in tag.find_all('a', class_='buy')]
        for item in title:
            print("Book: ",item.text.strip())
        for item in author:
            author = item.text.split("\n")
            print("AUTHOR: ",author[2])
        for item in price:
            if 'eBook' in item.text.strip():
                print("eBook PRICE: ", item.text.strip())
            else:
                print("PRICE: ", item.text.strip())
        print()

Output: Please download the file given below to view the extracted data as a result of executing the above code.

Click to Download and View The Output

Example 5: Scraping Using Relative Links

Until now we have seen examples where we scraped data directly from a webpage. Now, we will find out how we can extract data from websites that have hyperlinks. In this example, we shall extract data from https://codingbat.com/. Let us try and extract all the questions listed under the Python category in codingbat.com.

The demonstartion given below depicts a sample data that we are going to extract from the website.

Link to website: https://codingbat.com/python

Solution:

# 1. Import the necessary LIBRARIES
import requests
from bs4 import BeautifulSoup

# 2. Create a User Agent (Optional)
headers = {"User-Agent": "Mozilla/5.0 (Linux; U; Android 4.2.2; he-il; NEO-X5-116A Build/JDQ39) AppleWebKit/534.30 ("
                         "KHTML, like Gecko) Version/4.0 Safari/534.30"}

# 3. Send get() Request and fetch the webpage contents
response = requests.get('http://codingbat.com/python', headers=headers)
webpage = response.content

# 4. Check Status Code (Optional)
# print(response.status_code)

# 5. Create a Beautiful Soup Object
soup = BeautifulSoup(webpage, "html.parser")

# The Logic
url = 'https://codingbat.com'
div = soup.find_all('div', class_='summ')
links = [url + div.a['href'] for div in div]
for link in links:
    #print(link)
    second_page = requests.get(link, headers={
        "User-Agent": "Mozilla/5.0 (Linux; U; Android 4.2.2; he-il; NEO-X5-116A Build/JDQ39) AppleWebKit/534.30 ("
                      "KHTML, like Gecko) Version/4.0 Safari/534.30"})
    sub_soup = BeautifulSoup(second_page.content, 'html.parser')
    div = sub_soup.find('div', class_='tabc')
    question = [url + td.a['href'] for td in div.table.find_all('td')]

    for link in question:
        third_page = requests.get(link)
        third_soup = BeautifulSoup(third_page.content, 'html.parser')
        indent = third_soup.find('div', attrs={'class': 'indent'})
        problem = indent.table.div.string
        siblings_of_statement = indent.table.div.next_siblings
        demo = [sibling for sibling in siblings_of_statement if sibling.string is not None]
        print(problem)
        for example in demo:
            print(example)

        print("\n")

Output: Please download the file given below to view the extracted data as a result of executing the above code.

Click to Download and View Output

Conclusion

I hope you enjoyed the examples discussed in the article. Please subscribe and stay tuned for more articles and video contents in the future!

Where to Go From Here?

Enough theory, let’s get some practice!

To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

Practice projects is how you sharpen your saw in coding!

Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?

Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

Join my free webinar “How to Build Your High-Income Skill Python” and watch how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

The post Python BeautifulSoup Examples first appeared on Finxter.

Python

via Finxter https://ift.tt/2HRc2LV

November 24, 2020 at 08:55PM

Posted on

Raising Baby Yoda

Raising Baby Yoda

https://ift.tt/3pW7oh3

Raising Baby Yoda

Link

It might seem like zooming around the galaxy with Baby Yoda is all action and adventure. The Mandalorian’s Din Djarin explains how much work it is to take care of a 50-year-old poop machine in this clip that we’re sure many parents can relate to. A fun and charming fan parody from the guys at The Warp Zone.

fun

via The Awesomer https://theawesomer.com

November 24, 2020 at 06:30PM

Posted on

Initial Setup with Laravel Breeze

Initial Setup with Laravel Breeze

https://ift.tt/3nIOtEB


Your Ad Here!

Laracasts is a fantastic place to promote your development services to millions of visitors across the globe.

Let’s chat. This ad spot could be yours.

programming

via Laracasts https://ift.tt/1eZ1zac

November 20, 2020 at 03:19PM

Posted on

How to Add Stripe One-Time Payment Form to Laravel Project

How to Add Stripe One-Time Payment Form to Laravel Project

https://ift.tt/35JYmvE


Payments are one of the most typical elements of any web-project, and Stripe is a payment provider that is really easy to install in Laravel projects. In this article, we will add a payment form to the page.

As an example, we will take a Product Show page from our QuickAdminPanel Product Management module, but you can follow the same instructions and add Stripe form to ANY Laravel project page.

The plan will consist of 8 steps:

  1. Install Laravel Cashier
  2. Run Cashier migrations
  3. Stripe Credentials in .env
  4. User Model should be Billable
  5. Controller: Form Payment Intent
  6. Blade Page: Form, Styles, and Scripts
  7. Controller: Post Payment Processing
  8. After Successful Purchase: Send Product

Let’s begin!

1. Install Laravel Cashier

Run this command:

composer require laravel/cashier

Notice: Currently, the latest version of Cashier is v12. If you’re reading this article when the newer version has arrived, please read its upgrade guide. But personally, I doubt that any fundamentals will change.

2. Run Cashier migrations

Cashier package registers its own database migration directory, so remember to migrate your database after installing the package:

php artisan migrate

Those migrations are not in database/migrations folder, they are inside /vendor. Here are the contents.

1. Four new columns to users table:

Schema::table('users', function (Blueprint $table) {
    $table->string('stripe_id')->nullable()->index();
    $table->string('card_brand')->nullable();
    $table->string('card_last_four', 4)->nullable();
    $table->timestamp('trial_ends_at')->nullable();
});

2. New table subscriptions:

Schema::create('subscriptions', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->unsignedBigInteger('user_id');
    $table->string('name');
    $table->string('stripe_id');
    $table->string('stripe_status');
    $table->string('stripe_plan')->nullable();
    $table->integer('quantity')->nullable();
    $table->timestamp('trial_ends_at')->nullable();
    $table->timestamp('ends_at')->nullable();
    $table->timestamps();

    $table->index(['user_id', 'stripe_status']);
});

3. New table subscription_items:

Schema::create('subscription_items', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->unsignedBigInteger('subscription_id');
    $table->string('stripe_id')->index();
    $table->string('stripe_plan');
    $table->integer('quantity');
    $table->timestamps();

    $table->unique(['subscription_id', 'stripe_plan']);
});

3. Stripe Credentials in .env

There are two Stripe credentials that you need to add in your .env file:

STRIPE_KEY=pk_test_xxxxxxxxx
STRIPE_SECRET=sk_test_xxxxxxxxx

Where to get those “key” and “secret”? In your Stripe Dashboard:

Stripe Dashboard Laravel Keys env

Keep in mind, there are two “modes” of Stripe keys: testing and live keys. While on your local or testing servers, please remember to use TESTING keys, you can view them by toggling “View Testing Data” on the left menu:

Stripe view testing data

Another way to know if you’re using testing/live keys: the testing keys start with sk_test_ and pk_test_, and live keys start with sk_live_ and pk_live_. Also, live keys won’t work without SSL certificate enabled.

Notice: if you work in a team, when you add new variables, it’s a very good practice to also add them with empty values in .env.example. Then your teammates will know what variables are needed on their server. Read more here.

4. User Model should be Billable

Simple step: in your User model, add Billable trait from Cashier:

app/Models/User.php:

// ...
use Laravel\Cashier\Billable;

class User extends Authenticatable
{
    use HasFactory, Billable;

5. Controller: Form Payment Intent

To enable the Stripe payment form, we need to create a thing called “payment intent” and pass it to the Blade.

In this case, we will add it to ProductController method show():

class ProductController extends Controller
{
    // ...

    public function show(Product $product)
    {
        $intent = auth()->user()->createSetupIntent();

        return view('frontend.coupons.show', compact('product', 'intent'));
    }

Method createSetupIntent() comes from the Billable trait that we added just above in User model.

6. Blade Page: Form, Styles, and Scripts

This is the form that we will add from Stripe, with cardholder name, card number, expiry month/year, CVV code, and ZIP code.

Stripe payment form in Laravel

Luckily, Stripe documentation tells us exactly what HTML/JavaScript/CSS code should be added.

So, in our show.blade.php, we add this:

<form method="POST" action="" class="card-form mt-3 mb-3">
    @csrf
    <input type="hidden" name="payment_method" class="payment-method">
    <input class="StripeElement mb-3" name="card_holder_name" placeholder="Card holder name" required>
    <div class="col-lg-4 col-md-6">
        <div id="card-element"></div>
    </div>
    <div id="card-errors" role="alert"></div>
    <div class="form-group mt-3">
        <button type="submit" class="btn btn-primary pay">
            Purchase
        </button>
    </div>
</form>

All the input variables are exactly as Stripe suggests it, the only element that you would need to change is the route, where the form would be posted, so this:

route('products.purchase', $product->id)

We will create that route and Controller method in the next step.

Meanwhile, we also need to include Stripe’s Styles and JavaScript.
Let’s imagine that in your main Blade file, you have @yield sections for styles and scripts, like this:

<!DOCTYPE html>
<html>

<head>
    ...

    @yield('styles')
</head>

<body>
    ...

    
    @yield('scripts')
</body>
</html>

Then, in our show.blade.php, we may fill in those sections, with code from Stripe:

@section('styles')
<style>
    .StripeElement {
        box-sizing: border-box;
        height: 40px;
        padding: 10px 12px;
        border: 1px solid transparent;
        border-radius: 4px;
        background-color: white;
        box-shadow: 0 1px 3px 0 #e6ebf1;
        -webkit-transition: box-shadow 150ms ease;
        transition: box-shadow 150ms ease;
    }
    .StripeElement--focus {
        box-shadow: 0 1px 3px 0 #cfd7df;
    }
    .StripeElement--invalid {
        border-color: #fa755a;
    }
    .StripeElement--webkit-autofill {
        background-color: #fefde5 !important;
    }
</style>
@endsection

@section('scripts')
<script src="https://js.stripe.com/v3/"></script>
<script>
    let stripe = Stripe("")
    let elements = stripe.elements()
    let style = {
        base: {
            color: '#32325d',
            fontFamily: '"Helvetica Neue", Helvetica, sans-serif',
            fontSmoothing: 'antialiased',
            fontSize: '16px',
            '::placeholder': {
                color: '#aab7c4'
            }
        },
        invalid: {
            color: '#fa755a',
            iconColor: '#fa755a'
        }
    }
    let card = elements.create('card', {style: style})
    card.mount('#card-element')
    let paymentMethod = null
    $('.card-form').on('submit', function (e) {
        $('button.pay').attr('disabled', true)
        if (paymentMethod) {
            return true
        }
        stripe.confirmCardSetup(
            "",
            {
                payment_method: {
                    card: card,
                    billing_details: {name: $('.card_holder_name').val()}
                }
            }
        ).then(function (result) {
            if (result.error) {
                $('#card-errors').text(result.error.message)
                $('button.pay').removeAttr('disabled')
            } else {
                paymentMethod = result.setupIntent.payment_method
                $('.payment-method').val(paymentMethod)
                $('.card-form').submit()
            }
        })
        return false
    })
</script>

Inside of those sections, we’re adding two variables from the back-end:

env('STRIPE_KEY')

and

$intent->client_secret

So make sure you added them in the previous steps.

7. Controller: Post Payment Processing

Remember the route that we called in the previous step? Time to create it.

In routes/web.php, add this:

Route::post('products/{id}/purchase', 'ProductController@purchase')->name('products.purchase');

And then, let’s create a method in ProductController:

public function purchase(Request $request, Product $product)
{
    $user          = $request->user();
    $paymentMethod = $request->input('payment_method');

    try {
        $user->createOrGetStripeCustomer();
        $user->updateDefaultPaymentMethod($paymentMethod);
        $user->charge($product->price * 100, $paymentMethod);        
    } catch (\Exception $exception) {
        return back()->with('error', $exception->getMessage());
    }

    return back()->with('message', 'Product purchased successfully!');
}

So what is happening here?

1. We’re getting payment_method from the form (Stripe handles it in the background for us)
2. Then we call the Cashier methods to get/create the customer, set their payment method, and charge them.
3. Finally, we redirect back with success result
3b. If something goes wrong, try/catch block handles it and redirects back with an error.

Notice: variable $product->price is the price for your product, and we need to multiply it by 100 because Stripe charge is happening in cents.

To show the success message or errors, in your Blade file, you need to add something like this:

@if(session('message'))
    <div class="alert alert-success" role="alert"></div>
@endif
@if(session('error'))
    <div class="alert alert-danger" role="alert"></div>
@endif

8. After Successful Purchase: Send Product

After the customer paid for the product, you need to deliver the order. Of course, it depends on what they purchased and that code is very individual, but I will show you where to put it.

In fact, there are two ways. Easier but less secure, or harder and more secure.

Option 1. Fulfill Order in ProductController

You can do that directly in the same method:

public function purchase(Request $request, Product $product)
{
    $user          = $request->user();
    $paymentMethod = $request->input('payment_method');

    try {
        $user->createOrGetStripeCustomer();
        $user->updateDefaultPaymentMethod($paymentMethod);
        $user->charge($product->price * 100, $paymentMethod);        
    } catch (\Exception $exception) {
        return back()->with('error', $exception->getMessage());
    }

    // Here, complete the order, like, send a notification email
    $user->notify(new OrderProcessed($product)); 

    return back()->with('message', 'Product purchased successfully!');
}

Easy, right? The problem with that method is that it’s happening in sync, which means that $user->charge() may not be actually successfully finished, by the time you fulfill the order. In theory, it may cause false order deliveries with unsuccessful charges.

Option 2. Stripe Webhooks

Or, a more reliable method, is to catch so-called Stripe Webhooks. They ensure that the charge happened successfully, in the correct way. Whenever something happens in Stripe, they send a POST request to your server’s URL that you provide in the Stripe dashboard.

You can catch a lot of events from Stripe, and one of those events is charge.succeeded.

For that, I would recommend using a package called Laravel Stripe Webhooks, I’ve shot a separate video about it:

So if you want to catch more events, and not only charge success, I advise you to use Stripe Webhooks. Keep in mind they won’t (easily) work on your local computer, you need to set up a real domain that Stripe would call.

That’s it! Wish you to receive a lot of successful payments in your projects.

programming

via Laravel News Links https://ift.tt/2dvygAJ

November 19, 2020 at 08:57PM

Posted on

The Best Snow Shovel

The Best Snow Shovel

https://ift.tt/2Kq6M31

The Best Snow Shovel

We’ve been shoveling ourselves out of average snowstorms and heavy nor’easters with the True Temper 18-Inch Ergonomic Mountain Mover every winter since 2013. It’s the best snow shovel for most people looking to clear walkways, steps, and small driveways. No other shovel matches its unique blend of ideal size, ergonomics, durability, and availability.

technology

via Wirecutter: Reviews for the Real World https://ift.tt/36ATBn1

November 19, 2020 at 03:40PM

Posted on

Encrypt and Decrypt Data Using Keys

Encrypt and Decrypt Data Using Keys

https://ift.tt/3lNceuu

Crypto is a package by Spatie that allows you to easily generate public/private key pairs and then encrypt/decrypt messages using those keys.

The post Encrypt and Decrypt Data Using Keys appeared first on Laravel News.

Join the Laravel Newsletter to get Laravel articles like this directly in your inbox.

programming

via Laravel News https://ift.tt/14pzU0d

November 19, 2020 at 09:13AM

Posted on

Paracord Tutorial: Monkey Fist Stinger Impact Tool

Paracord Tutorial: Monkey Fist Stinger Impact Tool

https://ift.tt/3nCbuJq

(Image: Screenshot from video)

This is a pretty cool little tutorial on tying/weaving/braiding paracord to create a short length of cord with a handle on one end (complete with clip or key ring) and a paracord-encased steel ball on the other end. This “impact tool” can be called by many names, including monkey’s fist, mini-monkey fist, and stinger — and it looks as if, in the right hands, it could be an effective tool for self defense.

This is a good teaching video from “The Weavers of Eternity Paracord” and I have no doubt I could do this if I wished to… and I just might. I really like how some simple wraps can securely encase a steel sphere… pretty cool.

Ever make one of these, or some other paracord creation(s)? Let us know in the comments section.

The post Paracord Tutorial: Monkey Fist Stinger Impact Tool appeared first on AllOutdoor.com.

guns

via All Outdoor https://ift.tt/2yaNKUu

November 18, 2020 at 01:03PM