8 Ways to Stay on Top of the Latest Trends in Data Science

https://static1.makeuseofimages.com/wordpress/wp-content/uploads/2023/06/documents-on-wooden-surface.jpg

Data science is constantly evolving, with new papers and technologies coming out frequently. As such, data scientists may feel overwhelmed when trying to keep up with the latest innovations.

MUO video of the day

SCROLL TO CONTINUE WITH CONTENT

However, with the right tips, you can stay current and remain relevant in this competitive field. Thus, here are eight ways to stay on top of the latest trends in data science.

1. Follow Data Science Blogs and Newsletters

Data science blogs are a great way to brush up on the basics while learning about new ideas and technologies. Several tech conglomerates produce high-quality blog content where you can learn about their latest experiments, research, and projects. Great examples are Google, Facebook, and Netflix blogs, so waste no time checking them out.

Alternatively, you can look into online publications and individual newsletters. Depending on your experience level and advancement in the field, these blogs may address topics you’d find more relatable. For example, Version Control for Jupyter Notebook is easier for a beginner to digest than Google’s Preference learning for cache eviction.

You can find newsletters by doing a simple search, but we’d recommend Data Elixir, Data Science Weekly, and KDnuggets News, as these are some of the best.

2. Listen to Data Science Podcasts and Watch YouTube Videos

Podcasts are easily accessible and a great option when you’re pressed for time and want to get knowledge on the go. Listening to podcasts exposes you to new data science concepts while letting you carry out other activities simultaneously. Also, using interviews with experts in the field, some podcasts offer a window into the industry and let you learn from professionals’ experiences.

On the other hand, YouTube is a better alternative for audio-visual learners and has several videos at your disposal. Channels like Data School and StatQuest with Josh Starmer cover a wide range of topics for both aspiring and experienced data scientists. They also touch on new trends and methods, so following these channels is a good idea to keep current.

It’s easy to get lost in a sea of podcasts and videos, so carefully select detailed videos and the best podcasts for data science. This way, you can acquire accurate knowledge from the best creators and channels.

3. Learn Data Science Skills and Concepts From Courses and Books

Online courses allow learning from data science academics and experts, who condense their years of experience into digestible content. Recent courses cover several data science necessities, from hard-core machine learning to starting a career in data science without a degree. They may not be cheap, but they are well worth their cost in the value they give.

Additionally, books play an important role as well. Reading current data science books can help you learn new techniques, understand real-world data science applications, and develop critical thinking and problem-solving skills. These books explain in-depth data science concepts you may not find elsewhere.

Such books include The Data Science Handbook, Data Science on the Google Cloud Platform, and Think Bayes. You should also check out a few data science courses on sites like Coursera and Udemy.

4. Meet Industry Experts and Enthusiasts From Events and Communities

Attending conferences ushers you into an environment of like-minded individuals you can connect with. Although talking to strangers may feel uncomfortable, you will learn so much from the people at these events. By staying home, you will likely miss out on networking, job opportunities, and modern techniques like deep learning methods.

Furthermore, presentations allow you to observe other projects and familiarize yourself with the latest trends. Seeing what big tech companies are up to is encouraging and educative, and you can always take away something from them to apply in your work.

Data science events can be physical or virtual. Some good data science events to consider are the Open Data Science Conference (ODSC), Data Science Salon, and the Big Data and Analytics Summit.

5. Participate in Data Science Competitions and Hackathons

Data science hackathons unite data scientists to develop models that solve real-world problems within a specified time frame. They can be hosted by various platforms, such as Kaggle, DataHack, or UN Big Data Hackathon.

Participating in hackathons enhances your mastery and accuracy and exposes you to the latest data science tools and popular techniques for building models. Regardless of your results, competing with other data scientists in hackathons offers valuable insights into the latest advancements in data science.

Consider participating in the NERSC Open Hackathon, BNL Open Hackathon, and other virtual hackathons. Also, don’t forget to register for physical hackathons that may be happening near your location.

6. Contribute to Data Science Open Source or Social Good Projects

Contributing to open-source data science projects lets you work with other data scientists in development. From them, you’ll learn new tools and frameworks used by the data science community, and you can study project codes to implement in your work.

Furthermore, you can collaborate with other data scientists with different perspectives in an environment where exchanging ideas, feedback, and insights is encouraged. You can discover the latest techniques data science professionals use, industry standards, best practices, and how they keep up with data science trends.

First, search for repositories tagged with the data science topic on GitHub or Kaggle. Once you discover a project, consider how to contribute, regardless of your skill level, and start collaborating with other data scientists.

Following data science thought leaders and influencers on social media keep you informed about the latest data science trends. This way, you can learn about their views on existing subject matters and up-to-date news on data science trends. Additionally, it allows you to inquire about complicated subjects and get their reply.

You can take it a step further and follow Google, Facebook, Apple, and other big tech companies on Twitter. This gives you the privilege of knowing tech trends to expect, not only limited to data science.

Kirk Borne, Ronald van Loon, and Ian Goodfellow are some of the biggest names in the data science community. Start following them and big tech companies on Twitter and other social media sites to stay updated.

8. Share Your Data Science Work and Insights

Sharing your work lets you get feedback and suggestions from other data scientists with different experience levels and exposure. Their comments, questions, and critiques can help you stay up-to-date with the latest trends in data science.

You can discover trendy ideas, methods, tools, or resources you may not have known before by listening to their suggestions. For example, a person may unknowingly use an outdated version of Python until he posts his work online and someone points it out.

Sites like Kaggle and Discord have several data science groups through which you can share your work and learn. After signing up and joining a group, start asking questions and interacting with other data scientists. Prioritize knowledge, remember to be humble, and try to build mutually beneficial friendships with other data scientists.

Be a Lifelong Learner in Data Science

Continuous learning is necessary to remain valuable as a data scientist, but it can be difficult to keep up all by yourself. Consequently, you’ll need to find a suitable community to help you, and Discord is one of the best platforms to find one. Find a server with people in the same field, and continue your learning with your new team.

MakeUseOf

Laravel Migrations: “Table already exists” After Foreign Key Failed

https://laraveldaily.com/storage/437/Copy-of-Copy-of-Copy-of-ModelpreventLazyLoading();-(4).png

If you create foreign keys in your migrations, there may be a situation that the table is created successfully, but the foreign key fails. Then your migration is “half successful”, and if you re-run it after the fix, it will say “Table already exists”. What to do?


The Problem: Explained

First, let me explain the problem in detail. Here’s an example.

Schema::create('teams', function (Blueprint $table) {

$table->id();

$table->string('name');

$table->foreignId('team_league_id')->constrained();

$table->timestamps();

});

The code looks good, right? Now, what if the referenced table “team_leagues” doesn’t exist? Or maybe it’s called differently? Then you will see this error in the Terminal:

2023_06_05_143926_create_teams_table ..................................................................... 20ms FAIL

 

Illuminate\Database\QueryException

 

SQLSTATE[HY000]: General error: 1824 Failed to open the referenced table 'team_leagues'

(Connection: mysql, SQL: alter table `teams` add constraint `teams_team_league_id_foreign` foreign key (`team_league_id`) references `team_leagues` (`id`))

But that is only part of the problem. So ok, you realized that the referenced table is called “leagues” and not “team_leagues”. Possible fix options:

  • Either rename the field of “team_league_id” to just “league_id”
  • Or, specify the table ->constrained('leagues')

But the real problem now is the state of the database:

  • The table teams is already created
  • But the foreign key to leagues has failed!

This means there’s no record of this migration success in the “migrations” Laravel system DB table.

Now, the real problem: if you fix the error in the same migration and just run php artisan migrate, it will say, “Table already exists”.

2023_06_05_143926_create_teams_table ...................................................................... 3ms FAIL

 

Illuminate\Database\QueryException

 

SQLSTATE[42S01]: Base table or view already exists:

1050 Table 'teams' already exists

(Connection: mysql, SQL: create table `teams` (...)

So should you create a new migration? Rollback? Let me explain my favorite way of solving this.


Solution: Schema::hasTable() and Separate Foreign Key

You can re-run the migration for already existing tables and ensure they would be created only if they don’t exist with the Schema::hasTable() method.

But then, we need to split the foreignId() into parts because it’s actually a 2-in-1 method: it creates the column (which succeeded) and the foreign key (which failed).

So, we rewrite the migration into this:

if (! Schema::hasTable('teams')) {

Schema::create('teams', function (Blueprint $table) {

$table->id();

$table->string('name');

$table->unsignedBigInteger('team_league_id');

$table->timestamps();

});

}

 

// This may be in the same migration file or in a separate one

Schema::table('teams', function (Blueprint $table) {

$table->foreign('team_league_id')->constrained('leagues');

});

Now, if you run php artisan migrate, it will execute the complete migration(s) successfully.

Of course, an alternative solution would be to go and manually delete the teams table via SQL client and re-run the migration with the fix, but you don’t always have access to the database if it’s remote. Also, it’s not ideal to perform any manual operations with the database if you use migrations. It may be ok on your local database, but this solution above would be universal for any local/remote databases.

Laravel News Links

If The Mandalorian Was The A-Team

https://theawesomer.com/photos/2023/06/mandalorian_a_team_t.jpg

If The Mandalorian Was The A-Team

Link

The Mandalorian has some great action. So did The A-Team. So Nebulous Bee thought it would be fun to combine the two. This edit reimagines the credits for the Star Wars series in the style of the 1980s hit series, with Bo-Katan, Paz Vizsla, The Armorer, and Din Djarin standing in for Templeton Peck  B.A. Baracus, Hannibal Smith, and Howling Mad Murdock.

The Awesomer

JK Rowling hater demands she performs a sex act on him, but she ruins his life with a tweet instead

https://www.louderwithcrowder.com/media-library/image.png?id=34222424&width=980

JK Rowling has been the head transphobe in charge for quite some time now. You know how leftist sh*tc*nts get when you express a different opinion on them on the internet. They go right to the -isms and -phobias they accuse you of. In Rowling’s case, she also gets people threatening her life. All for the think-crime of, four years ago, defending someone who was fired for saying boys are boys and girls are girls.

If death threats don’t phase her, what do you think her putting her mouth on your weiner is going to do? One loser found this out the hard way and if he has any friends, I hope they’re all laughing at him now.,

So how did things escalate this quickly? It starts with Maya Forstater, the woman who got fired from her job for "’publishing “offensive” tweets questioning government proposals to allow people to self-identify as the opposite sex.’" Defending this woman is what started Rowling on her road to TERFdom. Forstater had just won a settlement when the organisation "was found to have engaged in unlawful discrimination in its decision not to offer her an employment contract or to renew her visiting fellowship."

Turns out that having a common belief about sex and gender does NOT equal bigotry.

Rowling offered Maya her congratulations.

Which led to Joshua D’silva telling Rowlings to suck his dick. This WAS the link to the tweet. It was literally deleted while I was working on this post after Rowling informed the world that D’silva allegedly had a penis so small, it was barely detectable. How embarrassing.

The whole Rowling thing still cracks me up. She is in no way, shape, or form on our side politically AT ALL. When leftists were "resisting" Trump in 2015-16, they would identify themselves as sects from Hogwarts. We would dare them to read another book.

All it took was one tweet and one single opinion for the left to turn on JK Rowling as if she was literally Voldermort. It’s hilarious. Especially knowing Rowling is sleeping on a giant pile of money as she responds. to each of her haters.

><><><><><><

Brodigan is Grand Poobah of this here website and when he isn’t writing words about things enjoys day drinking, pro-wrestling, and country music. You can find him on the Twitter too.

Facebook doesn’t want you reading this post or any others lately. Their algorithm hides our stories and shenanigans as best it can. The best way to stick it to Zuckerface? Bookmark LouderWithCrowder.com and check us out throughout the day! Also, follow us on Instagram and Twitter.

Louder With Crowder

Dynamic Database Config


README

Latest Version on Packagist
Quality Score
Code Quality
Vulnerability
Github Workflow Status
Total Downloads
Licence

This laravel package helps you dynamically set more database configurations through the .env file or database.

REQUIREMENTS

STEPS TO INSTALL

composer require ikechukwukalu/dynamicdatabaseconfig

Introduction

The need for this package came up when I once handled an already existing project that, due to certain constraints, had 9 databases implemented for each country their application was being utilised. This application also had a central database that was used by every country as well.

The config/database file wasn’t pretty. I’d prefer to have all configurations within the .env file only. The Big question was, what if the databases required grew to 19? These were the problems, both pending and existing that needed a clean hack/solution.

Middlewares

  • env.database.config
  • dynamic.database.config

Env.database.config Middleware

This middleware fetches database configurations from the .env file using postfixes like ONE. This dynamically declares an additional database connection for your laravel application.

DB_HOST_ONE=127.0.0.1
DB_PORT_ONE=3306
DB_DATABASE_ONE=second_db
DB_USERNAME_ONE=root
DB_PASSWORD_ONE=
  • Sample middleware implementation
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Route;

/**
 * mysql is the type of relational database connection being replicated - $database
 * mysql_1 is the new connection name - $name
 * ONE is the postfix - $postfix
 */

Route::middleware(['env.database.config:mysql,mysql_1,ONE'])->group(function () {
    Route::post('/user', function (Request $request) {
        /**
         * $request->_db_connection === 'mysql_1'
         */
        return \App\Models\User::on('mysql_1')->find(1);
    });
});

Route::post('/user', function (Request $request) {
        /**
         * $request->_db_connection === 'mysql_1'
         */
        return \App\Models\User::on('mysql_1')->find(1);
})->middleware('env.database.config:mysql,mysql_1,ONE');

You would not need to add a postfix, ONE, parameter to the middleware for the $postFix variable if you simply set the following session value session(config('dynamicdatabaseconfig.session_postfix')), but when a postfix parameter has been set, it will be used instead of the session value.

Dynamic.database.config Middleware

This middleware fetches database configurations from the database_configurations table within the primary migration database. It utilises a unique $ref variable. It’s recommended that the unique $ref variable should be human readable, that way it becomes easier to run the package’s console commands for running migrations. This will also dynamically declare an additional database connection for your laravel application.

use Ikechukwukalu\Dynamicdatabaseconfig\Models\DatabaseConfiguration;

protected $hidden = [
        'ref',
        'name',
        'database',

        /**
         * Accepts only arrays
         */
        'configuration'
];
  • Sample eloquent database insert
$countries = ['nigeria', 'ghana', 'togo', 'kenya'];
$config = \Config::get('database.connections.mysql');

foreach ($countries as $country) {
    $config['database'] = $country . '_db';
    DatabaseConfiguration::firstOrCreate(
    ['ref' => $country],
    [
        'ref' => $country,
        'name' => 'mysql_' . $country,
        'database' => 'mysql',
        'configuration' => $config
    ]);
}
  • Sample middleware implementation
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Route;

/**
 * nigeria is $ref value
 */

Route::middleware(['dynamic.database.config:nigeria'])->group(function () {
    Route::post('/user', function (Request $request) {
        /**
         * $request->_db_connection === 'mysql_nigeria'
         */
        return \App\Models\User::on('mysql_nigeria')->find(1);
    });
});

Route::post('/user', function (Request $request) {
        /**
         * $request->_db_connection === 'mysql_nigeria'
         */
        return \App\Models\User::on('mysql_nigeria')->find(1);
})->middleware('dynamic.database.config:nigeria');

You would not need to add a ref, nigeria, parameter to the middleware for the $ref variable if you simply set the following session value session(config('dynamicdatabaseconfig.session_ref')), but when a ref parameter has been set, it will be used instead of the session value.

By default, the values stored within the configuration field will be hashed, but you can adjust this from the .env file by setting DB_CONFIGURATIONS_HASH=false.

Migration

It’s compulsory to first migrate laravel’s initial database.

Other Migrations

  • Default migrations
  • Isolated migrations

Default Migrations

This will only migrate files within laravel’s default migration path database/migrations

php artisan env:migrate mysql mysql_1 ONE

php artisan dynamic:migrate nigeria

Isolated Migrations

This will only migrate files within the specified migration path database/migrations/folder

php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder

php artisan dynamic:migrate nigeria --path=database/migrations/folder

Both Migrations

Running the migrations as displayed below will result in the respective database having the migrated data from migrations within database/migrations and database/migrations/folder.

php artisan env:migrate mysql mysql_1 ONE
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder

php artisan dynamic:migrate nigeria
php artisan dynamic:migrate nigeria --path=database/migrations/folder

Database Seeding

php artisan env:migrate mysql mysql_1 ONE --seed
php artisan env:migrate mysql mysql_1 ONE --seeder=DatabaseSeederOne
php artisan env:migrate mysql mysql_1 ONE --seeder=DatabaseSeederOne  --path=database/migrations/folder

php artisan dynamic:migrate nigeria --seed
php artisan dynamic:migrate nigeria --seeder=DatabaseSeederNigeria
php artisan dynamic:migrate nigeria --seeder=DatabaseSeederNigeria  --path=database/migrations/folder

Re-runing Migrations Afresh

php artisan env:migrate mysql mysql_1 ONE --fresh
php artisan env:migrate mysql mysql_1 ONE --fresh --seed
php artisan env:migrate mysql mysql_1 ONE --fresh --seeder=DatabaseSeederOne
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder --fresh
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder --fresh --seeder=DatabaseSeederOne

php artisan dynamic:migrate nigeria --fresh
php artisan dynamic:migrate nigeria --fresh --seed
php artisan dynamic:migrate nigeria --fresh --seeder=DatabaseSeederNigeria
php artisan dynamic:migrate nigeria --path=database/migrations/folder --fresh
php artisan dynamic:migrate nigeria --path=database/migrations/folder --fresh --seeder=DatabaseSeederNigeria

Refreshing Migrations

php artisan env:migrate mysql mysql_1 ONE --refresh
php artisan env:migrate mysql mysql_1 ONE --refresh --seed
php artisan env:migrate mysql mysql_1 ONE --refresh --seeder=DatabaseSeederOne
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder --refresh
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder --refresh --seeder=DatabaseSeederOne

php artisan dynamic:migrate nigeria --refresh
php artisan dynamic:migrate nigeria --refresh --seed
php artisan dynamic:migrate nigeria --refresh --seeder=DatabaseSeederNigeria
php artisan dynamic:migrate nigeria --path=database/migrations/folder --refresh
php artisan dynamic:migrate nigeria --path=database/migrations/folder --refresh --seeder=DatabaseSeederNigeria

Rolling Back Migrations

php artisan env:migrate mysql mysql_1 ONE --rollback
php artisan env:migrate mysql mysql_1 ONE --path=database/migrations/folder --rollback

php artisan dynamic:migrate nigeria --rollback
php artisan dynamic:migrate nigeria --path=database/migrations/folder --rollback

NOTE

  • A primary database is needed before any other database can be migrated.
  • A database will be created if it does not exist.
  • Each database will retain it’s own independent migration table.
  • It’s recommended that you do not publish the package’s migration file, unless you want the database_configurations table to be migrated into every extra database created when running Default migrations.

PUBLISH MIGRATIONS

  • php artisan vendor:publish --tag=ddc-migrations

PUBLISH CONFIG

  • php artisan vendor:publish --tag=ddc-config

LICENSE

The DDC package is an open-sourced software licensed under the MIT license.

Laravel News Links

How to Store JSON Data in Database in Laravel (with example)

https://laracoding.com/wp-content/uploads/2023/06/how-to-store-json-data-in-database-in-laravel-with-example_841.png

Storing JSON data in a Laravel database provides a flexible solution for managing dynamic attributes or unstructured data. In this tutorial, we will walk through the process of storing JSON data in a Laravel database, using a practical example of storing product attributes. By the end of this tutorial, you will have a clear understanding of how to store and retrieve JSON data efficiently using Laravel.

Step 1: Setting up a Laravel Project

Create a new Laravel project using the following command:

laravel new json-data-storage

Step 2: Creating the Products Table Migration

Generate a migration file to create the products table using the command:

php artisan make:migration create_products_table --create=products

Inside the generated migration file (database/migrations/YYYY_MM_DD_create_products_table.php), define the table schema with a JSON column for the attributes:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    /**
     * Run the migrations.
     */
    public function up(): void
    {
        Schema::create('products', function (Blueprint $table) {
            $table->id();
            $table->json('attributes');
            $table->timestamps();
        });
    }

    /**
     * Reverse the migrations.
     */
    public function down(): void
    {
        Schema::dropIfExists('products');
    }
};

Step 3: Running the Migration

Run the migration using the following command:

php artisan migrate

Step 4: Creating the Product Model

Generate a Product model using the command:

php artisan make:model Product

In the Product model (app/Models/Product.php), add the $casts property to specify that the attributes attribute should be treated as JSON:

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Product extends Model
{
    use HasFactory;

    protected $casts = [
        'attributes' => 'json',
    ];
}

Step 5: Storing Product Attributes as JSON

To store product attributes as JSON, we can simply use Laravel’s Eloquent model. Create a new Product instance, set the desired attributes as an array, and save it:

use App\Models\Product;

$product = new Product;
$product->attributes = [
    'color' => 'red',
    'size' => 'medium',
    'weight' => 0.5,
];
$product->save();

Step 6: Retrieving JSON Data

To retrieve the attributes of a product, you can access the attributes property on the model:

$product = Product::find(1);
$attributes = $product->attributes;

If you want to access a specific attribute within the JSON data, you can do so by using array access:

$color = $product->attributes['color'];

By accessing the attributes property, you can retrieve the JSON data associated with the product and access specific attributes as needed.

Step 7: Manipulating JSON Data (Full Array)

To update all the attributes of the product at once, you can assign a new array with all the key-value pairs and use Eloquent’s save() method to update the record directly.

$product = \App\Models\Product::find(1);
$product->attributes = [
    'color' => 'green',
    'size' => 'large',
    'weight' => 2.5,
];
$product->save();

Step 8: Manipulating JSON Data (One Value)

Updating a single value within the JSON data requires a slightly different approach in Laravel and has one important caveat. Directly modifying the attribute like $product->attributes['weight'] = 1.0 and saving the product will result in an ErrorException: “Indirect modification of overloaded property App\Models\Product::$attributes has no effect.”

To overcome this issue, you can follow the solution below:

$product = \App\Models\Product::find(1);
$attributes = $product->attributes; // create a copy of the array
$attributes['weight'] = 0.6; // modify the value in the copied array
$product->attributes = $attributes; // assign the copied array back to $product->attributes
$product->save();

Conclusion

Storing JSON data in a database using Laravel provides flexibility and convenience when working with dynamic or unstructured data. By following the steps outlined in this tutorial, you have learned how to create the necessary migrations and models, store and retrieve JSON data, and manipulate the JSON data efficiently. This knowledge equips you with the tools to handle various use cases in your Laravel applications and opens up possibilities for efficiently managing complex data structures.

Start implementing JSON data storage in your Laravel projects and unlock the full potential of your application’s data management capabilities. Happy coding!

Laravel News Links

Data School: Make your own *private* GPT with Python 🔒

https://www.dataschool.io/content/images/2023/06/ai.jpegMake your own *private* GPT with Python ????

ChatGPT is amazing, but its knowledge is limited to the data on which it was trained.

Wouldn&apost it be great if you could use the power of Large Language Models (LLMs) to interact with your own private documents, without uploading them to the web?

The great news is that you can do this TODAY! Let me show you how…

privateGPT is an open source project that allows you to parse your own documents and interact with them using a LLM. You ask it questions, and the LLM will generate answers from your documents.

All using Python, all 100% private, all 100% free!

Below, I&aposll walk you through how to set it up. (Note that this will require some familiarity with the command line.)


1️⃣ Clone or download the repository

If git is installed on your computer, then navigate to an appropriate folder (perhaps "Documents") and clone the repository (git clone https://github.com/imartinez/privateGPT.git). That will create a "privateGPT" folder, so change into that folder (cd privateGPT).

Alternatively, you could download the repository as a zip file (using the green "Code" button), move the zip file to an appropriate folder, and then unzip it. It will create a folder called "privateGPT-main", which you should rename to "privateGPT". You&aposll then need to navigate to that folder using the command line.


2️⃣ Create and activate a new environment

I highly recommend setting up a virtual environment for this project. My tool of choice is conda, which is available through Anaconda (the full distribution) or Miniconda (a minimal installer), though many other tools are available.

If you&aposre using conda, create an environment called "gpt" that includes the latest version of Python using conda create -n gpt python. Then, activate the environment using conda activate gpt. Use conda list to see which packages are installed in this environment.

(Note: privateGPT requires Python 3.10 or later.)


3️⃣ Install the packages listed in requirements.txt

First, make sure that "privateGPT" is your working directory using pwd. Then, make sure that "gpt" is your active environment using conda info.

Once you&aposve done that, use pip3 install -r requirements.txt to install all of the packages listed in that file into the "gpt" environment. This will take at least a few minutes.

Use conda list to see the updated list of which packages are installed.

(Note: The System Requirements section of the README may be helpful if you run into an installation error.)


4️⃣ Download the LLM model

In the Environment Setup section of the README, there&aposs a link to an LLM. Currently, that LLM is ggml-gpt4all-j-v1.3-groovy.bin. Download that file (3.5 GB).

Then, create a subfolder of the "privateGPT" folder called "models", and move the downloaded LLM file to "models".


5️⃣ Copy the environment file

In the "privateGPT" folder, there&aposs a file named "example.env". Make a copy of that file named ".env" using cp example.env .env. Use ls -a to check that it worked.

(Note: This file has nothing to do with your virtual environment.)


6️⃣ Add your documents

Add your private documents to the "source_documents" folder, which is a subfolder of the "privateGPT" folder. Here&aposs a list of the supported file types.

I recommend starting with a small number of documents so that you can quickly verify that the entire process works. (The "source_documents" folder already contains a sample document, "state_of_the_union.txt", so you can actually just start with this document if you like.)


7️⃣ Ingest your documents

Once again, make sure that "privateGPT" is your working directory using pwd.

Then, run python ingest.py to parse the documents. This may run quickly (< 1 minute) if you only added a few small documents, but it can take a very long time with larger documents.

Once this process is done, you&aposll notice that there&aposs a new subfolder of "privateGPT" called "db".


8️⃣ Interact with your documents

Run python privateGPT.py to start querying your documents! Once it has loaded, you will see the text Enter a query:

Type in your question and hit enter. After a minute, it will answer your question, followed by a list of source documents that it used for context.

(Keep in mind that the LLM has "knowledge" far outside your documents, so it can answer questions that have nothing to do with the documents you provided to it.)

When you&aposre done asking questions, just type exit.


???? Troubleshooting

This project is less than two months old, and it depends on other libraries which are also quite new! Thus it&aposs highly likely that you will run into bugs, unexplained errors, and crashes.

For example, if you get an "unknown token" error after asking a question, my experience has been that you can ignore the error and you will still get an answer to your question.

On the other hand, if you get a memory-related error, you will need to end the process by hitting "Ctrl + C" on your keyboard. (Then, just restart it by running python privateGPT.py.)

You might be able to find a workaround to a particular problem by searching the Issues in the privateGPT repository.

If you post your own GitHub issue, please be kind! This is an open source project being run by one person in his spare time (for free)!


???? Usage tips

Want to query more documents? Add them to the "source_documents" folder and re-run python ingest.py.

Want to start over? Delete the "db" folder, and a new "db" folder will be created the next time you ingest documents.

Want to hide the source documents for each answer? Run python privateGPT.py -S instead of python privateGPT.py.

Want to try a different LLM? Download a different LLM to the "models" folder and reference it in the ".env" file.

Want to use the latest version of the code? Given its popularity, it&aposs likely that this project will evolve rapidly. If you want to use the latest version of the code, run git pull origin main.


???? Disclaimer

I think it&aposs worth repeating the disclaimer listed at the bottom of the repository:

This is a test project to validate the feasibility of a fully private solution for question answering using LLMs and Vector embeddings. It is not production-ready, and it is not meant to be used in production. The models selection is not optimized for performance, but for privacy; but it is possible to use different models and vector stores to improve performance.

Planet Python

This Rig Turns Walls/Windows Into Animated Lite-Brites

https://s3files.core77.com/blog/images/1414093_81_124043_JrOgOXAJJ.jpg

Interior/exterior decoration, 2020s-style: Govee Curtain Lights are essentially a curtain of hanging LED-embedded strips. Measuring 1.5m (5′) wide and 2m (6.6′) tall, this provides a grid of 520 evenly-spaced LEDs that you can program via an app, allowing you to turn walls or windows into gigantic animated Lite-Brites.

You can either choose from stock animations provided by the company, or create your own. The company also boasts of a "Music Mode" that makes the lights move in accordance with music.

If this is your jam, startup Govee paid the YouTuber below to produce this video on setting up and using the product.

All I can think is how dated this is going to look in a few years; I imagine if this style of decoration catches on, the consumer will demand a much higher level of resolution.

Core77