This used to happen before big releases or when something was broken in production and I could not figure it out. Now it is something else. It is the quiet thought that shows up after all the noise fades.

AI can write code.

Not toy examples. Real code. Code that looks like something I would have written after years of doing this job.

I am a senior software engineer. I have spent years getting here. Late nights, messy systems, strange bugs that made no sense until they suddenly did. That experience used to feel like a moat. Lately, it feels thinner.

For a while, I leaned on that experience. I told myself there were things AI would never understand: the weird decisions buried inside old systems, the half-broken logic that only exists because something else depends on it, the kind of knowledge you only get from being there when things went wrong.

The Ground is Moving

Then I started seeing things that did not fit that story. I watched an AI refactor a block of code I had been avoiding for days. It was not perfect, but it was clean and readable and better than what was there. A few days later, it pointed out a subtle concurrency bug. The kind that usually takes time and patience to track down. It caught it in seconds.

That was the moment something shifted for me. The machine does not “understand” the way we do. But it does not need to. With enough context, it can trace patterns across a codebase and arrive at something that works. It can hold more of the system in its head than I ever could. And it does not get tired.

That is the part that keeps me up sometimes. Not fear in a dramatic sense. Just a quiet realization that the ground under the role is moving. We are not competing with tools anymore. We are standing next to systems that can generate more output in a minute than we can review in an hour.

The math is different now.

Where Experience Still Matters

For a while, I kept asking myself the same question: If the machine can write the code, what exactly am I being paid for?

That question forced me to look at my work in a way I had not before. I do not get paid to type fast. I do not get paid to remember syntax. The value has never really been in the act of writing code. It just looked that way because that was the visible part.

The value is in deciding what should be built. In knowing when something is wrong even if it compiles. In understanding the tradeoffs that are not written anywhere. In taking responsibility when things break. Those are the moments where the cost of being wrong is high. That is where experience still matters.

A modern commercial jet flies almost entirely on autopilot, but we still pay pilots a premium to handle that 1% of pure uncertainty. Software development is rapidly shifting toward this exact model.

I started changing how I work. I let the AI handle the parts I used to grind through: setting up endpoints, filling in repetitive logic, getting a rough draft of something on the screen. I use it to scan through files when I have a vague idea of where a bug might be hiding. It is fast. It clears space.

Directing the Process

The interesting part is what happens after that. I spend more time deciding if the feature should exist at all. I look at how it fits into the system instead of just how to implement it. When something breaks, I am not just fixing a function. I am tracing how data flows through multiple services and figuring out where the assumptions failed.

There is also a new kind of work that did not exist before. Guiding the AI. Breaking a vague problem into something it can handle. Knowing when to trust the output and when to throw it away. It feels less like writing code and more like directing a process.

That shift took some time to accept. It felt like I was letting go of something I had worked hard to get good at. But the alternative is worse: holding on to a version of the job that is slowly disappearing.

The Right Side of the Line

The role is changing whether we like it or not. Some parts of software development will shrink. The kind of work that is repetitive and predictable is already being absorbed by these tools. There is no point pretending otherwise.

What remains is harder. You need to see the system as a whole. You need to make decisions with incomplete information. You need to deal with people, not just code. You need to take responsibility for outcomes that go beyond a single pull request. That is where the job is moving.

I still wake up at 2 AM sometimes.

The feeling is different now. It is not just worry. It is curiosity mixed with a bit of pressure. There is a version of this field where the people who adapt get to build things faster and at a scale that was not possible before.

We don't need 10x engineers anymore. We just need 1x engineers with 10x leverage.

There is also a version where you get left behind because you kept doing things the old way. I am trying to stay on the right side of that line.

The tools have changed. The expectations have changed. The core question has not:

What problems are worth solving, and how do you solve them well?

C++ is a versatile and powerful programming language that was developed as an extension of the C programming language. It combines procedural, object-oriented, and generic programming paradigms, making it suitable for a wide range of applications. C++ allows programmers to write efficient and high-performance code, making it a popular choice for system programming, game development, embedded systems, and more.

Brief History and Evolution of C++

C++ has a fascinating history that spans several decades. Let's take a brief look at its evolution:

1. Origins in C: In the early 1970s, Dennis Ritchie developed the C programming language at Bell Labs. C quickly gained popularity due to its simplicity and portability, becoming a lingua franca for software development.

2. C with Classes: In the late 1970s, Bjarne Stroustrup, a Danish computer scientist, began working on extending the C language to support object-oriented programming. The result was "C with Classes," which introduced the concept of classes, derived classes, and other fundamental aspects of OOP.

3. C++ Emerges: In 1983, Stroustrup renamed "C with Classes" to C++, signifying its evolutionary nature. The "++" symbol represents an increment, indicating improvements over the C language. Stroustrup added features like virtual functions, function overloading, and operator overloading, cementing C++ as a distinct language.

4. Standardization Efforts: As C++ gained popularity, the need for a standardized version became evident. The first C++ standard, known as C++98 or C++03, was published in 1998. Subsequent revisions, including C++11, C++14, C++17, and C++20, introduced numerous enhancements and modern features to the language.

5. Features and Advancements: Over the years, C++ has evolved to provide a wide range of features, including templates, exceptions, namespaces, the Standard Template Library (STL), lambdas, and more. These advancements have significantly enhanced the expressiveness, productivity, and performance of C++ programs.

6. C++ in Modern Times: Today, C++ continues to be widely used and actively developed. The language maintains compatibility with C, allowing seamless integration with existing C codebases. It also offers extensive support across different platforms and compilers.

C++ is a powerful programming language that builds upon the foundation of C while introducing key concepts of object-oriented programming. Its evolution from C with Classes to a standardized language has solidified its position as a versatile choice for developing complex software systems. In the next sections of this article, we will delve deeper into the syntax, features, and best practices of C++, empowering you to leverage its potential to the fullest.

Setting up CLion IDE

CLion is a powerful Integrated Development Environment (IDE) specifically designed for C++ development. In this article, we will guide you through the steps to set up CLion, so you can start writing C++ code efficiently. By the end of this guide, you'll have CLion installed and configured for your C++ projects.

1. Installing CLion:

   • Visit the JetBrains CLion website (https://www.jetbrains.com/clion/) and download the appropriate version for your operating system.
   • Run the installer and follow the on-screen instructions to complete the installation process.
   • Launch CLion after the installation is complete.

2. Configuring CLion:

   • On the welcome screen, you can select "Do not import settings" if this is your first time using CLion or choose to import settings if you have a previous configuration you'd like to bring in.
   • Set up the Keymap preferences by selecting your preferred keymap scheme or customizing the key bindings according to your needs.
   • Configure the Appearance settings, including the theme, fonts, and colour scheme, to your liking.

3. Creating a New C++ Project:

   • Click on "Create New Project" or select "New Project" from the "File" menu.
   • In the project template selection window, choose "C++ Executable" or any other relevant template that suits your project requirements.
   • Specify the project name, location, and other necessary details. Click "Create" to create the project.

4. Setting Up the Compiler:

   • CLion typically comes bundled with JetBrains' custom C++ compiler, which provides a seamless development experience.
   • If you prefer using a different compiler, such as GCC or Clang, you can configure it in CLion's settings:
     • Go to "File" -> "Settings" (or "Preferences" on macOS).
     • Navigate to "Build, Execution, Deployment" -> "Toolchains".
     • Click on the "+" button to add a new toolchain and select your preferred compiler.
     • Provide the necessary details and paths for the compiler installation and click "OK".

5. Configuring the Build System:

   • CLion utilizes the CMake build system by default. CMake allows for easy project configuration and generation of build files.
   • CLion automatically detects CMakeLists.txt files in your project directory and configures the build system accordingly.
   • Modify the CMakeLists.txt file as per your project requirements, such as adding source files, libraries, and compiler flags.

6. Building and Running a C++ Program:

   • Open the main C++ source file in the CLion editor.
   • To build the project, click on the "Build" button or use the "Build" -> "Build Project" option.
   • Once the build process completes without errors, you can run the program by clicking on the "Run" button or using the "Run" -> "Run" option.
   • CLion provides a built-in debugger to help you with debugging your C++ programs. You can set breakpoints, step through code, and inspect variables during runtime.

Basic Syntax and Structure

Understanding the basic syntax and structure of a programming language is fundamental to getting started with any programming endeavour. In this article, we will explore the core elements of C++, including the Hello World program, data types and variables, operators and expressions, control structures, and functions. By the end of this guide, you will have a solid foundation in C++ syntax and be able to write simple programs.

Hello World program in C++

The traditional Hello World program is a simple yet essential starting point for learning any programming language. In C++, it looks like this:

#include <iostream>

int main() {
    std::cout << "Hello, World!" << std::endl;
    return 0;
}

• Explanation:
  • #include <iostream>: This line includes the input/output stream library, which allows us to work with input and output operations in C++.
  • int main(): Every C++ program must have a main() function as its entry point.
  • std::cout << "Hello, World!" << std::endl;: The std::cout statement is used to display output to the console. In this case, it prints "Hello, World!".
  • return 0;: The return statement ends the main() function and returns a value (0 in this case) to indicate successful execution.

Data types and variables

C++ provides several built-in data types to represent different kinds of values. Here are some commonly used data types:

• int: Represents integer values.
• float and double: Represent floating-point (decimal) values with single and double precision, respectively.
• char: Represents a single character.
• bool: Represents boolean values (true or false).

Variables are used to store and manipulate data. Here's an example of declaring and using variables:

int age = 25;
float pi = 3.14159;
char grade = 'A';
bool isStudent = true;

• Explanation:
  • The int, float, char, and bool keywords are used to declare variables of the corresponding data types.
  • The = symbol is used for initialization, assigning a value to a variable.
  • Variables can be named according to your preference, but they must follow certain naming conventions and avoid reserved keywords.

Operators and expressions

Operators allow us to perform various operations on variables and values. Here are some common operators in C++:

• Arithmetic operators: +, ,, /, % (addition, subtraction, multiplication, division, modulo)
• Relational operators: ==, !=, <, >, <=, >= (equality, inequality, less than, greater than, less than or equal to, greater than or equal to)
• Logical operators: &&, ||, ! (logical AND, logical OR, logical NOT)
• Assignment operator: =
• Increment and decrement operators: ++, -

Expressions combine variables, values, and operators to produce results. Here's an example:

int x = 5;
int y = 10;
int z = x + y;  // z will be 15

1. Control structures (if-else, loops): Control structures allow us to control the flow of execution in a program. Here are two common control structures in C++:

2. if-else statement:

     int x = 5;
     if (x > 10) {
         std::cout << "x is greater than 10" << std::endl;
     } else {
         std::cout << "x is less than or equal to 10" << std::endl;
     }
   

3. loops (for loop):

     for (int i = 1; i <= 5; i++) {
         std::cout << i << " ";
     }
     // Output: 1 2 3 4 5
   

4. Explanation:

   • In the if-else statement, the condition inside the parentheses determines which block of code to execute based on the result.
   • The for loop consists of three parts: initialization, condition, and increment. It repeatedly executes the code block until the condition becomes false.

Functions and procedures

Functions are reusable blocks of code that perform specific tasks. They are essential for organizing and modularizing code. Here's an example:

int add(int a, int b) {
    return a + b;
}

int result = add(5, 3);
std::cout << "Result: " << result << std::endl;

• Explanation:
  • The add() function takes two integer parameters (a and b) and returns their sum.
  • The result variable stores the returned value from the add() function call.
  • The std::cout statement displays the result on the console.

Object-Oriented Programming (OOP) Concepts

Object-Oriented Programming (OOP) is a powerful paradigm that allows us to organize code in a modular and reusable manner. In this article, we will explore key OOP concepts in C++, including classes and objects, encapsulation, inheritance, polymorphism, constructors and destructors, access specifiers, and function overloading and overriding. Through code snippets and explanations, we will gain a solid understanding of these fundamental concepts.

Classes and Objects:

• Classes serve as blueprints for creating objects in C++.
• A class defines the properties (data members) and behaviors (member functions) of an object.
• Here's an example of a simple class in C++:

class Rectangle {
    int width;
    int height;

public:
    void setDimensions(int w, int h) {
        width = w;
        height = h;
    }

    int calculateArea() {
        return width * height;
    }
};

Encapsulation, Inheritance, and Polymorphism:

• Encapsulation is the practice of bundling data and the functions that operate on that data into a single unit (a class).
• Inheritance allows the creation of new classes (derived classes) from existing ones (base classes), inheriting their properties and behaviors.
• Polymorphism enables objects of different classes to be treated as objects of a common base class, facilitating code reusability and flexibility.
• Here's an example demonstrating these concepts:

class Animal {
public:
    virtual void makeSound() {
        cout << "The animal makes a sound.\\n";
    }
};

class Dog : public Animal {
public:
    void makeSound() override {
        cout << "The dog barks.\\n";
    }
};

class Cat : public Animal {
public:
    void makeSound() override {
        cout << "The cat meows.\\n";
    }
};

Constructors and Destructors:

• Constructors are special member functions used to initialize objects of a class.
• Destructors are used to perform cleanup tasks when an object goes out of scope or is explicitly destroyed.
• Here's an example showing the usage of constructors and destructors:

class Student {
    string name;
    int age;

public:
    // Constructor
    Student(string n, int a) {
        name = n;
        age = a;
        cout << "Student object created.\\n";
    }

    // Destructor
    ~Student() {
        cout << "Student object destroyed.\\n";
    }
};

Access Specifiers (Public, Private, Protected):

• Access specifiers determine the visibility and accessibility of class members.
• Public members are accessible from anywhere in the program.
• Private members can only be accessed within the class.
• Protected members are accessible within the class and its derived classes.
• Here's an example illustrating access specifiers:

class Car {
private:
    int speed;

public:
    void setSpeed(int s) {
        speed = s;
    }

    int getSpeed() {
        return speed;
    }
};

Function Overloading and Overriding:

• Function overloading allows multiple functions with the same name but different parameters.
• Function overriding occurs when a derived class provides its own implementation of a function defined in the base class.
• Here's an example showcasing function overloading and overriding:

class Shape {
public:
    void calculateArea() {
        cout << "Calculating area of a generic shape.\\n";
    }

    void calculateArea(int length, int width) {
        cout << "Calculating area of a rectangle: " << length * width << endl;
    }
};

class Circle : public Shape {
public:
    void calculateArea() override {
        cout << "Calculating area of a circle.\\n";
    }
};

Pointers and Memory Management

Memory management is a critical aspect of C++ programming, and understanding pointers is essential for efficient memory handling. In this article, we will explore pointers and addresses, dynamic memory allocation using 'new' and delete, common memory leaks, and how to avoid them. We will also introduce smart pointers as a safer alternative for memory management in C++.

Pointers and Addresses

Pointers are variables that store memory addresses. They allow us to manipulate data indirectly and efficiently. Let's see a simple example:

#include <iostream>

int main() {
    int x = 10; // Declare an integer variable
    int* ptr = &x; // Declare a pointer and initialize it with the address of 'x'

    // Output the value and address of 'x' using the pointer
    std::cout << "Value of x: " << *ptr << std::endl;
    std::cout << "Address of x: " << ptr << std::endl;

    return 0;
}

In this code snippet, we declare an integer variable x and a pointer ptr that holds the address of x. We then use the pointer to access the value of x using the dereference operator (*ptr).

Dynamic Memory Allocation (new and delete)

C++ provides operators new and delete for dynamic memory allocation. Unlike stack memory (local variables), dynamic memory is allocated on the heap, and its lifespan is determined by the programmer.

#include <iostream>

int main() {
    int* arr = new int[5]; // Dynamically allocate an integer array of size 5

    // Initialize the array elements
    for (int i = 0; i < 5; ++i) {
        arr[i] = i + 1;
    }

    // Output the array elements
    for (int i = 0; i < 5; ++i) {
        std::cout << arr[i] << " ";
    }

    // Don't forget to deallocate the memory to avoid memory leaks
    delete[] arr;

    return 0;
}

In this example, we use new to allocate memory for an integer array of size 5. We then initialize and output the array elements. After using the array, it is crucial to deallocate the memory using delete[] to prevent memory leaks.

Memory Leaks and How to Avoid Them

Memory leaks occur when allocated memory is not deallocated, leading to a waste of memory over time. It is essential to be cautious when using dynamic memory.

Consider the following code snippet:

#include <iostream>

int main() {
    int* num = new int; // Allocate memory for a single integer

    // Perform some operations with 'num'
    *num = 42;
    std::cout << "Value of num: " << *num << std::endl;

    // Oops! Forgot to deallocate the memory.
    // delete num; // Uncomment this line to fix the memory leak

    return 0;
}

In this example, we allocate memory for a single integer using new. However, we forget to deallocate the memory using delete. This results in a memory leak.

To avoid memory leaks, always ensure that you release the allocated memory using delete (or delete[] for arrays) when you are done using it.

Smart Pointers in C++

Smart pointers are a valuable feature introduced in C++11 to simplify memory management and prevent memory leaks. They provide automatic memory deallocation when the smart pointer goes out of scope, eliminating the need for manual memory management using delete. There are three types of smart pointers in C++: unique_ptr, shared_ptr, and weak_ptr.

unique_ptr: std::unique_ptr provides exclusive ownership of the dynamically allocated memory. It ensures that only one unique_ptr can own the memory at a time, making it an excellent choice for cases where exclusive ownership is required.

#include <iostream>
#include <memory>

int main() {
    std::unique_ptr<int> num = std::make_unique<int>(42);

    // Perform some operations with 'num'
    std::cout << "Value of num: " << *num << std::endl;

    // No need to delete explicitly; memory is automatically deallocated

    return 0;
}

In this example, we create a unique_ptr named num that points to an integer with a value of 42. The memory is deallocated automatically when num goes out of scope, ensuring proper cleanup.

shared_ptr: std::shared_ptr allows multiple smart pointers to share ownership of the same dynamically allocated object. It keeps track of the number of references to the object and automatically deallocates the memory when the last 'shared_ptr' that references it is destroyed.

#include <iostream>
#include <memory>

int main() {
    std::shared_ptr<int> num1 = std::make_shared<int>(42);
    std::shared_ptr<int> num2 = num1;

    // Perform operations with 'num1' and 'num2'
    std::cout << "Value of num1: " << *num1 << std::endl;
    std::cout << "Value of num2: " << *num2 << std::endl;

    // Memory is deallocated when all shared_ptrs referencing it are destroyed

    return 0;
}

In this example, we create two shared_ptr instances, num1 and num2, both pointing to the same integer object. The memory is deallocated automatically when both num1 and num2 go out of scope, ensuring proper cleanup.

weak_ptr: std::weak_ptr is a type of smart pointer that provides a non-owning reference to an object managed by shared_ptr. It allows you to observe or access the object without affecting its lifetime. A weak_ptr does not contribute to the reference count, and it is useful when you want to avoid cyclic references.

#include <iostream>
#include <memory>

int main() {
    std::shared_ptr<int> sharedNum = std::make_shared<int>(42);
    std::weak_ptr<int> weakNum = sharedNum;

    // Check if the object still exists before accessing it
    if (auto lockedNum = weakNum.lock()) {
        std::cout << "Value of sharedNum through weakNum: " << *lockedNum << std::endl;
    } else {
        std::cout << "Object has been destroyed." << std::endl;
    }

    // Memory is deallocated when all shared_ptrs referencing it are destroyed
return 0;
}

In this example, we create a shared_ptr named sharedNum and a weak_ptr named weakNum that refers to the same integer object. We use the lock() function to obtain a temporary shared_ptr from weakNum and check if the object is still valid before accessing it.

Understanding the Standard Template Library (STL)

The Standard Template Library (STL) is an essential part of C++ that provides a collection of template classes and functions to simplify common programming tasks. In this article, we will explore the key components of the STL, including containers (vectors, lists, maps, etc.), iterators and algorithms, string manipulation, and input/output streams (iostream). We'll dive into code snippets to illustrate how to use these components effectively in your C++ programs.

Overview of STL Containers

The STL offers various container classes that provide data structures for holding and managing collections of elements efficiently. Some popular STL containers include:

Vector:

#include <vector>

std::vector<int> numbers = {1, 2, 3, 4, 5};

• Vectors are dynamic arrays that can grow or shrink in size automatically.
• They provide random access to elements and support dynamic resizing with efficient memory management.

List:

#include <list>

std::list<std::string> names = {"Alice", "Bob", "Charlie"};

• Lists are doubly-linked lists that allow fast insertions and deletions from any position.
• They do not support random access and indexing like vectors, but they are efficient for certain operations.

Map:

#include <map>

std::map<std::string, int> scores = {{"Alice", 95}, {"Bob", 88}, {"Charlie", 92}};

• Maps are associative containers that store key-value pairs.
• They are implemented as binary search trees, providing efficient key-based retrieval and insertion.

Iterators and Algorithms

STL iterators are objects that allow iterating over elements in containers without exposing the underlying data structure's details. Algorithms are functions that operate on container elements through iterators. Some commonly used algorithms include:

Finding an element in a vector:

#include <algorithm>

auto it = std::find(numbers.begin(), numbers.end(), 3);
if (it != numbers.end()) {
    std::cout << "Found element: " << *it << std::endl;
} else {
    std::cout << "Element not found." << std::endl;
}

• The std::find algorithm searches for the first occurrence of a value in a range.

Sorting a vector in ascending order:

#include <algorithm>

std::sort(numbers.begin(), numbers.end());

• The std::sort algorithm arranges elements in ascending order using the < operator by default.

String Manipulation:

The STL provides various functions for string manipulation, including concatenation, substring extraction, and searching. Some examples are:

Concatenating strings:

#include <string>

std::string first_name = "John";
std::string last_name = "Doe";
std::string full_name = first_name + " " + last_name;

• The + operator can be used to concatenate strings.

Extracting substrings:

std::string sentence = "The quick brown fox";
std::string word = sentence.substr(4, 5); // Extract "quick"

• The substr function extracts a substring from a given position with a specified length.

Input/Output Streams (iostream): C++ uses the iostream library to perform input and output operations. It provides the cin and cout objects for reading from and writing to the standard input and output streams, respectively. Here's an example:

#include <iostream>

int main() {
    int number;
    std::cout << "Enter a number: ";
    std::cin >> number;
    std::cout << "You entered: " << number << std::endl;
    return 0;
}

• The std::cin object is used with the >> operator to read input from the user.
• The std::cout object is used with the << operator to print output to the console.

Exception Handling

Exception handling is a crucial aspect of C++ programming that allows us to gracefully handle errors and unexpected situations that may occur during program execution. In this article, we will explore how to handle errors and exceptions in C++ using try-catch blocks, learn about throwing and catching exceptions, and understand the importance of exception safety and resource management.

Handling Errors and Exceptions in C++

In C++, when an error occurs during program execution, it can lead to unexpected behavior or even program crashes. Exception handling provides a mechanism to handle these errors gracefully, ensuring that the program can recover from exceptional conditions.

An exception is an object that represents an exceptional condition, such as division by zero, file not found, or memory allocation failure. When an exception is thrown, the program searches for an appropriate exception handler to deal with the situation.

try-catch Blocks

C++ uses try-catch blocks to handle exceptions. The code inside the try block is monitored for exceptions. If an exception is thrown within the try block, the program immediately jumps to the corresponding catch block to handle the exception.

Syntax of a try-catch block:

try {
    // Code that may throw an exception
} catch (ExceptionType1& ex) {
    // Handle ExceptionType1
} catch (ExceptionType2& ex) {
    // Handle ExceptionType2
} // ...

The catch blocks are evaluated in the order they appear. The first catch block that matches the type of the thrown exception is executed.

Throwing and Catching Exceptions

In C++, exceptions are thrown using the throw keyword, followed by an object representing the exception. The object can be of any type, but it is commonly an instance of a specific exception class derived from std::exception.

Example of throwing and catching an exception:

#include <iostream>
#include <stdexcept>

double divide(int numerator, int denominator) {
    if (denominator == 0) {
        throw std::runtime_error("Division by zero is not allowed.");
    }
    return static_cast<double>(numerator) / denominator;
}

int main() {
    try {
        int result = divide(10, 0);
        std::cout << "Result: " << result << std::endl;
    } catch (const std::exception& ex) {
        std::cout << "Exception caught: " << ex.what() << std::endl;
    }
    return 0;
}

Exception Safety and Resource Management

Exception safety is a critical concern when dealing with resource management in C++. When an exception is thrown, it's essential to ensure that resources (memory, file handles, etc.) are properly cleaned up to avoid leaks and maintain a consistent program state.

• Use Resource Managing Classes: To manage resources safely, use C++ classes such as smart pointers (std::shared_ptr, std::unique_ptr) for memory management and RAII (Resource Acquisition Is Initialization) for other resources like file handles.
• Avoid Dynamic Memory Management: Whenever possible, prefer using containers and C++ Standard Library classes to manage data instead of manually allocating and deallocating memory.
• Use the Copy-and-Swap Idiom: In situations where an object's state can change during an operation, use the Copy-and-Swap Idiom to ensure exception safety.

Best Practices and Coding Guidelines

In the world of programming, adhering to best practices and coding guidelines is crucial for writing maintainable, efficient, and bug-free code. In this article, we will explore some essential best practices and coding guidelines specific to C++. By following these practices, you can enhance the readability, reliability, and performance of your C++ codebase.

Naming Conventions and Code Style:

• Consistent and meaningful naming conventions for variables, functions, classes, and constants.
• Using descriptive names that reflect the purpose and functionality of the entities.
• Following camel case, snake case, or any other agreed-upon convention for naming.
• Avoiding single-letter or ambiguous names that can lead to confusion.

Example:

// Good naming convention for variables and functions
int numberOfStudents;
void calculateTotalScore();

// Poor naming convention
int n;
void calc();

Error Handling and Exception Safety:

• Properly handling and reporting errors to maintain code stability and prevent unexpected behavior.
• Using exceptions to handle exceptional situations, rather than relying solely on return codes or error flags.
• Employing RAII (Resource Acquisition Is Initialization) principle to ensure proper cleanup and resource deallocation in case of exceptions.

Example:

try {
    // Code that may throw an exception
} catch (const std::exception& e) {
    // Handle the exception
    std::cerr << "An error occurred: " << e.what() << std::endl;
}

Memory Management and Resource Acquisition:

• Using smart pointers (e.g., std::unique_ptr, std::shared_ptr) to handle memory allocation and deallocation automatically.
• Avoiding raw pointers whenever possible to minimize the risk of memory leaks and dangling pointers.
• Following the principle of ownership to clearly define responsibility for resource allocation and deallocation.

Example:

std::unique_ptr<int> ptr = std::make_unique<int>(42);
// Automatically releases memory when ptr goes out of scope

Performance Considerations:

• Avoiding unnecessary copies and expensive operations by using move semantics and rvalue references.
• Utilizing algorithms and data structures from the Standard Template Library (STL) for efficient and optimized code.
• Profiling and optimizing critical sections of the codebase based on performance analysis.

Example:

std::string name = "John";
std::string lastName = std::move(name);  // Efficiently moves the content of name to lastName

C++ Coding Standards:

• Familiarizing yourself with established coding standards, such as the Google C++ Style Guide, to maintain a consistent code style across projects.
• Adhering to guidelines for indentation, spacing, commenting, and other formatting conventions.
• Using static code analysis tools (e.g., Clang-Tidy) to automatically enforce coding standards and detect potential issues.

Example (Google C++ Style Guide):

// Good code style with proper indentation and spacing
if (condition) {
    doSomething();
} else {
    doSomethingElse();
}

// Poor code style with inconsistent indentation and spacing
if(condition){
    doSomething();
}else{
    doSomethingElse();
}

Conclusion

In this comprehensive introduction to C++, we have covered a wide range of topics to help you embark on your journey into the world of C++ programming. Starting from the setting up of the development environment, including installing a C++ compiler, choosing an Integrated Development Environment (IDE), and configuring the build system, we have ensured that you have a solid foundation to begin writing C++ code.

We then delved into best practices and coding guidelines specific to C++. By following these guidelines, you can write code that is not only readable and maintainable but also robust and efficient. We explored the importance of naming conventions and code style, emphasizing the significance of clear and descriptive names to enhance code clarity. Additionally, we discussed error handling and exception safety, emphasizing the use of exceptions for handling exceptional situations and the adoption of the RAII principle for resource management.

Memory management and resource acquisition were also key topics covered in this article. By utilizing smart pointers and following the principle of ownership, we can ensure proper memory allocation and deallocation, minimizing the risk of memory leaks and dangling pointers.

Performance considerations were not overlooked, as we discussed techniques such as move semantics and rvalue references to optimize code and avoid unnecessary copies. Leveraging the Standard Template Library (STL) and profiling critical code sections can lead to highly efficient and optimized C++ programs.

Lastly, we explored the importance of adhering to established coding standards, such as the Google C++ Style Guide. Following these standards ensures consistency in code style and readability across projects, and utilizing static code analysis tools can automatically enforce these guidelines.

Armed with this knowledge, you are now equipped to tackle C++ programming projects with confidence. Remember, practice and continuous learning are essential to master any programming language, and C++ is no exception. As you progress in your journey, dive deeper into advanced C++ features and explore additional resources, such as books, online tutorials, and the vibrant C++ community.

In conclusion, C++ is a powerful and versatile programming language widely used in various domains, ranging from systems programming to game development. By understanding the fundamentals, setting up your development environment, and following best practices and coding guidelines, you are well on your way to becoming a proficient C++ developer. So, roll up your sleeves, start coding, and embrace the endless possibilities that C++ has to offer.

Redis, which stands for Remote Dictionary Server, is an open-source, in-memory data structure store that serves as a highly efficient and versatile database solution. Developed by Salvatore Sanfilippo, Redis is designed to offer lightning-fast data access and retrieval by keeping all the data in memory rather than on disk, ensuring remarkable performance and responsiveness.

One of the key features that sets Redis apart is its support for various data structures, including strings, lists, sets, sorted sets, and hashes. This versatility enables Redis to serve multiple purposes, such as acting as a key-value store, a message broker, or a caching layer.

Redis utilizes an in-memory approach, making it exceptionally fast for read and write operations. With its optimized data structures and efficient algorithms, Redis can process data in sub-millisecond response times, making it an ideal choice for real-time applications and use cases that require high-performance data manipulation.

Furthermore, Redis offers persistence options, allowing users to store their data on disk periodically or whenever specific conditions are met. This feature ensures data durability and makes Redis suitable for applications that require both speed and data persistence.

In addition to its exceptional performance, Redis provides advanced features like pub/sub messaging, transactions, and Lua scripting, which empower developers to build complex applications and systems with ease.

Redis has become immensely popular due to its simplicity, versatility, and extensive support for multiple programming languages. It offers client libraries for various languages, making it accessible and easy to integrate into different software stacks.

Whether you need to cache frequently accessed data, build real-time applications, implement a messaging system, or store and process complex data structures, Redis provides a robust and efficient solution that continues to be widely adopted across industries.

In conclusion, Redis is a powerful and flexible database solution that excels in performance, versatility, and ease of use. Its in-memory nature, support for various data structures, and extensive feature set make it a go-to choice for developers looking to build high-performance applications and systems.

How Redis is different from other Databases?

Redis differs from other databases in several key aspects:

1. In-Memory Data Storage: Redis primarily stores data in memory, allowing for extremely fast read and write operations. This design choice sets Redis apart from traditional disk-based databases that incur disk I/O overhead. By keeping data in memory, Redis can achieve high throughput and low latency, making it well-suited for applications that require real-time data processing and high-performance caching.
2. Data Structures: Redis supports a wide range of data structures, including strings, lists, sets, hashes, and sorted sets. This flexibility allows developers to model complex data scenarios efficiently. Unlike many other databases that offer a fixed schema, Redis provides dynamic and versatile data structures that can be manipulated and combined to suit specific application needs.
3. Built-in Caching Mechanisms: Redis has native support for caching, making it an excellent choice for scenarios that require frequent access to frequently accessed data. With features like key expiration and eviction policies, Redis can efficiently manage cache storage and ensure that the most relevant data remains readily available.
4. Pub/Sub Messaging: Redis includes built-in publish/subscribe messaging capabilities. This feature allows applications to implement real-time messaging and event-driven architectures. Publishers can send messages to specific channels, and subscribers receive messages from those channels in real-time. This makes Redis a suitable choice for building chat systems, real-time analytics, and other applications that require event-based communication.
5. Lua Scripting: Redis supports Lua scripting, which enables developers to execute server-side scripts directly within the database. This allows for complex operations and transactional logic to be performed within Redis, reducing the need for round-trips between the application and the database. Lua scripting enhances Redis's flexibility and extensibility by enabling custom behaviour and complex data manipulations.
6. Replication and Persistence: Redis provides built-in replication mechanisms for creating multiple copies of the database. This replication ensures high availability and fault tolerance by propagating changes from a master node to one or more slave nodes. Redis also offers persistence options through snapshotting and append-only file (AOF) logging, allowing data to be saved to disk for recovery in the event of a system failure.
7. Simplicity and Performance: Redis is known for its simplicity and performance. It offers a straightforward set of commands that are easy to understand and use. The focus on in-memory storage and the absence of complex query languages or indexing mechanisms simplify the database's design and contribute to its exceptional performance.

Overall, Redis stands out among other databases due to its in-memory storage, versatile data structures, caching capabilities, pub/sub messaging, Lua scripting, replication, and performance-oriented design. These features make Redis a popular choice for applications that require speed, flexibility, and real-time data processing.

Installing Redis and Redis CLI

Here are the steps to install Redis on Linux, macOS, and Windows:

Linux:

1. Open a terminal window.

2. Update the package manager:

    sudo apt update
   

3. Install Redis:

    sudo apt install redis-server
   

4. Start the Redis service:

    sudo systemctl start redis-server
   

5. To access the Redis command-line interface (redis-cli):

    redis-cli
   

macOS:

1. Open a terminal window.

2. Install Homebrew (if not already installed):

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

3. Install Redis using Homebrew:

    brew install redis
   

4. Start the Redis service:

    brew services start redis
   

5. To access the Redis command-line interface (redis-cli):

    redis-cli
   

Windows:

1. Download the latest Redis for Windows from the Microsoft Open Tech GitHub repository: https://github.com/microsoftarchive/redis/releases
2. Extract the downloaded archive to a directory of your choice, e.g., C:\Redis.
3. Open a Command Prompt window.

4. Navigate to the Redis installation directory:

    cd C:\Redis
   

5. Start the Redis server:

    redis-server.exe
   

6. Open another Command Prompt window to access the Redis command-line interface (redis-cli):

    cd C:\Redis
    redis-cli.exe
   

Please note that for Linux and macOS, the installation steps assume the usage of package managers like apt and Homebrew, respectively. If you are using a different package manager or distribution, you may need to adjust the installation commands accordingly.

Commands

SET manipulation

1. SET

   • Command: SET
   • Syntax: SET key value
   • Usage: SET myset "Hello Redis"
   • Description: Sets the value of a key with the provided string value. If the key already exists, it overwrites the existing value. This command is also used to create a new key-value pair if the key doesn't exist.

2. GET

   • Command: GET
   • Syntax: GET key
   • Usage: GET myset
   • Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.

3. SADD

   • Command: SADD
   • Syntax: SADD key member [member ...]
   • Usage: SADD myset "value1" "value2"
   • Description: Adds one or more members to a set. If the members already exist, they are ignored. If the key doesn't exist, a new set is created with the specified members.

4. SMEMBERS

   • Command: SMEMBERS
   • Syntax: SMEMBERS key
   • Usage: SMEMBERS myset
   • Description: Returns all the members of a set. If the key doesn't exist, an empty set is returned.

5. SISMEMBER

   • Command: SISMEMBER
   • Syntax: SISMEMBER key member
   • Usage: SISMEMBER myset "value1"
   • Description: Checks if a member exists in a set. Returns 1 if the member is present; otherwise, returns 0.

6. SREM

   • Command: SREM
   • Syntax: SREM key member [member ...]
   • Usage: SREM myset "value1" "value2"
   • Description: Removes one or more members from a set. If the members don't exist, they are ignored. If the key becomes empty after removing members, the key is also removed.

7. SCARD

   • Command: SCARD
   • Syntax: SCARD key
   • Usage: SCARD myset
   • Description: Returns the cardinality (number of members) of a set. If the key doesn't exist, 0 is returned.

8. SINTER

   • Command: SINTER
   • Syntax: SINTER key [key ...]
   • Usage: SINTER myset1 myset2
   • Description: Returns the intersection of multiple sets. Only the members that exist in all the specified sets are included in the result set.

9. SUNION

   • Command: SUNION
   • Syntax: SUNION key [key ...]
   • Usage: SUNION myset1 myset2
   • Description: Returns the union of multiple sets. All the members from the specified sets are included in the result set.

10. SDIFF

    • Command: SDIFF
    • Syntax: SDIFF key [key ...]
    • Usage: SDIFF myset1 myset2
    • Description: Returns the difference between two sets. The members that exist in the first set but not in any other specified sets are included in the result set.

Sorted Sets manipulation

1. ZADD

   • Command: ZADD
   • Syntax: ZADD key [NX|XX] [CH] [INCR] score member [score member ...]
   • Usage: ZADD mysortedset 1 "value1" 2 "value2"
   • Description: Adds one or more members with scores to a sorted set. If a member already exists, its score is updated. Additional options like NX (only add new elements), XX (only update existing elements), CH (return the number of changed elements), and INCR (increment the score of existing member) can be specified.

2. ZSCORE

   • Command: ZSCORE
   • Syntax: ZSCORE key member
   • Usage: ZSCORE mysortedset "value1"
   • Description: Returns the score of a member in a sorted set. If the member doesn't exist, nil is returned.

3. ZRANGE

   • Command: ZRANGE
   • Syntax: ZRANGE key start stop [WITHSCORES]
   • Usage: ZRANGE mysortedset 0 -1
   • Description: Returns a range of members from a sorted set based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.

4. ZREVRANGE

   • Command: ZREVRANGE
   • Syntax: ZREVRANGE key start stop [WITHSCORES]
   • Usage: ZREVRANGE mysortedset 0 -1 WITHSCORES
   • Description: Returns a range of members from a sorted set in reverse order based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.

5. ZREM

   • Command: ZREM
   • Syntax: ZREM key member [member ...]
   • Usage: ZREM mysortedset "value1" "value2"
   • Description: Removes one or more members from a sorted set. If a member doesn't exist, it is ignored. If the key becomes empty after removing members, the key is also removed.

6. ZCARD

   • Command: ZCARD
   • Syntax: ZCARD key
   • Usage: ZCARD mysortedset
   • Description: Returns the cardinality (number of members) of a sorted set. If the key doesn't exist, 0 is returned.

7. ZCOUNT

   • Command: ZCOUNT
   • Syntax: ZCOUNT key min max
   • Usage: ZCOUNT mysortedset 1 10
   • Description: Returns the count of members in a sorted set within the specified score range (min and max inclusive).

8. ZRANK

   • Command: ZRANK
   • Syntax: ZRANK key member
   • Usage: ZRANK mysortedset "value1"
   • Description: Returns the position (rank) of a member in a sorted set when sorted in ascending order. The lowest rank is 0. If the member doesn't exist, nil is returned.

9. ZREVRANK

   • Command: ZREVRANK
   • Syntax: ZREVRANK key member
   • Usage: ZREVRANK mysortedset "value1"
   • Description: Returns the position (rank) of a member in a sorted set when sorted in descending order. The highest rank is 0. If the member doesn't exist, nil is returned.

10. ZINCRBY

    • Command: ZINCRBY
    • Syntax: ZINCRBY key increment member
    • Usage: ZINCRBY mysortedset 5 "value1"
    • Description: Increments the score of a member in a sorted set by the specified increment. If the member doesn't exist, it is added with the initial score as the increment value.

String, Number and Bit Manipulation

1. APPEND

   • Command: APPEND
   • Syntax: APPEND key value
   • Usage: APPEND mykey "world"
   • Description: Appends the specified value to the string value already stored at the key. If the key doesn't exist, a new key is created with the specified value.

2. BITCOUNT

   • Command: BITCOUNT
   • Syntax: BITCOUNT key [start end]
   • Usage: BITCOUNT mykey
   • Description: Counts the number of set bits (1s) in a string value. Optionally, a range can be specified to count the bits within that range.

3. BITOP

   • Command: BITOP
   • Syntax: BITOP operation destkey key [key ...]
   • Usage: BITOP AND destkey key1 key2
   • Description: Performs a bitwise operation (AND, OR, XOR, NOT) between multiple keys and stores the result in the destination key.

4. BITPOS

   • Command: BITPOS
   • Syntax: BITPOS key bit [start] [end]
   • Usage: BITPOS mykey 1 0 10
   • Description: Finds the position of the first set bit (1) or clear bit (0) in a string value. Optionally, a range can be specified to search for the bit within that range.

5. DECR

   • Command: DECR
   • Syntax: DECR key
   • Usage: DECR mykey
   • Description: Decrements the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before decrementing.

6. DECRBY

   • Command: DECRBY
   • Syntax: DECRBY key decrement
   • Usage: DECRBY mykey 5
   • Description: Decrements the value of the key by the specified decrement. If the key doesn't exist, it is initialized with the value of 0 before decrementing.

7. GET

   • Command: GET
   • Syntax: GET key
   • Usage: GET mykey
   • Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.

8. GETBIT

   • Command: GETBIT
   • Syntax: GETBIT key offset
   • Usage: GETBIT mykey 5
   • Description: Returns the bit value at the specified offset in a string value. The offset is zero-based.

9. GETRANGE

   • Command: GETRANGE
   • Syntax: GETRANGE key start end
   • Usage: GETRANGE mykey 0 3
   • Description: Retrieves a substring of the string value stored at the key, starting from the specified start index to the end index (inclusive).

10. GETSET

    • Command: GETSET
    • Syntax: GETSET key value
    • Usage: GETSET mykey "newvalue"
    • Description: Sets the value of the key with the provided string value and returns the old value. If the key doesn't exist, it is initialized with the specified value.

11. INCR

    • Command: INCR
    • Syntax: INCR key
    • Usage: INCR mykey
    • Description: Increments the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before incrementing.

12. INCRBY

    • Command: INCRBY
    • Syntax: INCRBY key increment
    • Usage: INCRBY mykey 5
    • Description: Increments the value of the key by the specified increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.

13. INCRBYFLOAT

    • Command: INCRBYFLOAT
    • Syntax: INCRBYFLOAT key increment
    • Usage: INCRBYFLOAT mykey 2.5
    • Description: Increments the value of the key by the specified float increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.

14. MGET

    • Command: MGET
    • Syntax: MGET key [key ...]
    • Usage: MGET key1 key2 key3
    • Description: Retrieves the values associated with multiple keys. It returns an array of values corresponding to the provided keys.

15. MSET

    • Command: MSET
    • Syntax: MSET key value [key value ...]
    • Usage: MSET key1 "value1" key2 "value2"
    • Description: Sets multiple keys to their respective values. It allows setting multiple key-value pairs in a single command.

16. MSETNX

    • Command: MSETNX
    • Syntax: MSETNX key value [key value ...]
    • Usage: MSETNX key1 "value1" key2 "value2"
    • Description: Sets multiple keys to their respective values, only if none of the keys already exist. It allows atomic setting of multiple keys.

17. PSETEX

    • Command: PSETEX
    • Syntax: PSETEX key milliseconds value
    • Usage: PSETEX mykey 1000 "Hello"
    • Description: Sets the value of the key with the provided string value and a specified expiration time in milliseconds.

18. SET

    • Command: SET
    • Syntax: SET key value [EX seconds|PX milliseconds] [NX|XX]
    • Usage: SET mykey "Hello" EX 60 NX
    • Description: Sets the value of a key with the provided string value. Additional options like expiration time, conditional set (NX - only if the key doesn't exist, XX - only if the key exists) can be specified.

19. SETBIT

    • Command: SETBIT
    • Syntax: SETBIT key offset value
    • Usage: SETBIT mykey 5 1
    • Description: Sets the bit at the specified offset in a string value to either 0 or 1.

20. SETEX

    • Command: SETEX
    • Syntax: SETEX key seconds value
    • Usage: SETEX mykey 60 "Hello"
    • Description: Sets the value of the key with the provided string value and a specified expiration time in seconds.

21. SETNX

    • Command: SETNX
    • Syntax: SETNX key value
    • Usage: SETNX mykey "Hello"
    • Description: Sets the value of a key with the provided string value, only if the key doesn't exist.

22. SETRANGE

    • Command: SETRANGE
    • Syntax: SETRANGE key offset value
    • Usage: SETRANGE mykey 5 "Redis"
    • Description: Overwrites a substring of the string value stored at the key, starting from the specified offset with the provided value.

23. STRLEN

    • Command: STRLEN
    • Syntax: STRLEN key
    • Usage: STRLEN mykey
    • Description: Returns length

Conclusion

In conclusion, we have discussed Redis, a popular in-memory data structure store, and explored its unique characteristics that set it apart from other databases. Redis offers exceptional performance and versatility, making it suitable for various use cases, such as caching, real-time analytics, messaging systems, and more.

We have also covered the installation process of Redis on Linux, macOS, and Windows, including how to access the Redis command-line interface (redis-cli) for interacting with the database.

Furthermore, we have delved into the major commands related to set manipulation, providing syntax, usage examples, and descriptions for each command. These commands enable users to perform various operations like adding, retrieving, modifying, and removing elements from sets in Redis.

Additionally, we have explored the major commands related to sorted set manipulation, including operations such as adding members with scores, retrieving ranges, finding positions, incrementing scores, and more. Sorted sets in Redis provide an ordered collection of unique elements, allowing efficient operations based on both element values and scores.

Redis commands like APPEND, GET, SET, INCR, and many others offer powerful functionality for string manipulation, bit operations, and key-value management.

Redis, with its versatility, performance, and rich set of commands, continues to be a preferred choice for developers and architects when it comes to handling data-intensive applications and solving various data storage and retrieval challenges.

In this article, we will take a look at the major announcements made at Google I/O 2023. We will discuss the new hardware products, software features, and cloud computing services that were announced. We will also discuss the potential benefits of these new technologies.

AI Updates

Google also announced a number of new AI and machine learning features. These include a new AI language model called PaLM 2, which is capable of generating text, translating languages, and writing different kinds of creative content. Google also announced a new AI-powered search feature called Search Generative Experience, which allows users to generate different creative text formats of text content, like poems, code, scripts, musical pieces, email, letters, etc.

PaLM2

PaLM 2 is a new AI language model that is built on top of Google's Pathways system. Pathways is a new AI architecture that is designed to be more efficient and scalable than previous AI architectures. PaLM 2 is trained on a massive dataset of text and code, and it can generate text, translate languages, and write different kinds of creative content.

Search Generative Experience

Search Generative Experience is a new AI-powered search feature that allows users to generate different creative text formats of text content, like poems, code, scripts, musical pieces, email, letters, etc. Search Generative Experience uses PaLM 2 to generate text content based on a user's query. For example, if a user searches for "poem about love," Search Generative Experience will generate a poem about love.

The new AI and machine learning features announced at I/O 2023 show that Google is continuing to invest heavily in these technologies. These new features have the potential to revolutionize the way we interact with computers and the way we access information.

Here are some of the potential benefits of the new AI and machine learning features announced at I/O 2023:

• More personalized experiences: AI and machine learning can be used to personalize experiences for users. For example, Search Generative Experience can generate text content that is tailored to a user's interests.

• More efficient workflows: AI and machine learning can be used to automate tasks and make workflows more efficient. For example, PaLM 2 can be used to generate code, which can save developers time and effort.

• New creative possibilities: AI and machine learning can be used to create new creative possibilities. For example, Search Generative Experience can be used to generate poems, code, scripts, musical pieces, email, letters, etc.

Project Tailwind

Google's Project Tailwind is an AI-powered notebook that learns from your documents. It can summarize your notes, generate a study guide, and answer natural language questions. Tailwind is still in development, but it has the potential to be a powerful tool for students, writers, and anyone who deals with a lot of text in their life.

To use Tailwind, you first need to pick files from your Google Drive. Tailwind then creates a private AI model with expertise in that information. You can then ask Tailwind questions about your notes, and it will provide answers based on its model. Tailwind can also summarize your notes, generate a study guide, and answer natural language questions.

Tailwind is currently in beta and is only available in the United States. You can join the waitlist to try it out here: https://thoughtful.withgoogle.com/about

Cloud Computing Updates

Google also announced a number of new cloud computing features. These include a new A3 supercomputer virtual machine, which is designed for resource-intensive workloads. Google also announced a number of new features for Google Cloud Platform, such as a new machine learning API and a new container orchestration service.

A3 Supercomputer Virtual Machine

The A3 supercomputer virtual machine is a new Google Cloud Platform offering that is designed for resource-intensive workloads. The A3 VM is powered by Nvidia A100 GPUs, and it offers up to 480GB of memory and 1.6TB of storage. The A3 VM is ideal for workloads such as machine learning, artificial intelligence, and high-performance computing.

New Machine Learning API

Google announced new generative AI cloud capabilities that open the door for developers with all different skill levels to build enterprise-ready applications. These capabilities include Imagen, a new image generation and customization model; Codey, a new code generation model; and Chirp, a universal speech model that brings speech-to-text accuracy to 100+ languages.

Duet AI for Google Cloud: Google also announced Duet AI for Google Cloud, an AI-powered collaborator that helps with contextual code completion, generates functions in real time, offers suggestions tuned to your code base, assists with code reviews and more.

New Safe Browsing API: Google announced a new Safe Browsing API that uses AI to identify and alert you to dangerous sites and files, stopping scams before they happen.

Expanded spam protections in Google Drive: Google is expanding spam protections in Google Drive with a new view that makes it easier to review files and decide what you might view as spam.

Expanded access to dark web report: Google is expanding access to its dark web report over the next few weeks so anyone with a Gmail account in the U.S. can scan to see if their Gmail address appears on the dark web.

New Container Orchestration Service

Google also announced a new container orchestration service at I/O 2023. The new service makes it easier for developers to deploy and manage containers. The service offers a number of features, such as automatic scaling, load balancing, and fault tolerance.

The new cloud computing features announced at I/O 2023 show that Google is continuing to invest heavily in cloud computing. These new features make it easier for developers to build and deploy applications that are scalable, reliable, and secure.

Here are some of the potential benefits of the new cloud computing features announced at I/O 2023:

• Increased scalability: The new A3 supercomputer virtual machine can handle even the most resource-intensive workloads.

• Improved reliability: The new container orchestration service makes it easier to deploy and manage containers, which can help to improve the reliability of applications.

• Enhanced security: The new machine learning API can help to improve the security of applications by making it easier to identify and prevent attacks.

New Software

Google also announced a number of new software features. These include Android 14, Google Maps Immersive View, and Google Assistant improvements.

Android 14

Android 14 is the next major version of Android, and it includes a number of new features, such as a new design, privacy improvements, and new productivity features.

The new design for Android 14 is based on Material You, and it includes a number of new features, such as new colors, new widgets, and new animations.

The privacy improvements in Android 14 include a new privacy dashboard, which allows users to see how their data is being used by apps. Android 14 also includes a new feature called "privacy indicators," which shows users when an app is accessing their microphone or camera.

The productivity features in Android 14 include a new multitasking menu, which makes it easier for users to switch between apps. Android 14 also includes a new feature called "desk clock," which allows users to use their phone as a second monitor.

Google Maps Immersive View

Google Maps Immersive View is a new feature that allows users to see a more realistic view of a location before they visit it. Immersive View uses 3D imagery and artificial intelligence to create a realistic representation of a location. This can be helpful for users who are planning a trip or who are trying to decide where to eat.

Google Assistant Improvements

Google Assistant is getting a number of new features, such as the ability to translate languages in real time and the ability to control smart home devices with your voice.

The ability to translate languages in real time is a major new feature for Google Assistant. This will allow users to have conversations with people who speak other languages. The ability to control smart home devices with your voice is another major new feature for Google Assistant. This will allow users to control their smart home devices without having to touch their phone or tablet.

New hardware

In addition to announcing new software products at I/O 2023, Google announced a number of new hardware products at I/O 2023, including the Pixel Fold, the Pixel Tablet, and the Pixel 7A. The Pixel Fold is Google's first foldable phone, and it features a 7.6-inch OLED display when unfolded. The Pixel Tablet is Google's first tablet in years, and it will run on Android 13. The Pixel 7A is a budget-friendly phone that will be available later this year.

The Pixel Fold

The Pixel Fold is Google's first foldable phone, and it features a 7.6-inch OLED display when unfolded. The phone is powered by Google's Tensor G2 chip and has a triple-lens rear camera system. The Pixel Fold is expected to be available later this year.

The Pixel Tablet

The Pixel Tablet is Google's first tablet in years, and it will run on Android 13. The tablet has a 10.95-inch display and is powered by Google's Tensor G2 chip. The Pixel Tablet is expected to be available later this year.

The Pixel 7A

The Pixel 7A is a budget-friendly phone that will be available later this year. The phone has a 6.1-inch OLED display and is powered by Google's Tensor G2 chip. The Pixel 7A also has a dual-lens rear camera system.

The new hardware products announced at I/O 2023 show that Google is continuing to innovate and expand its product portfolio. The Pixel Fold and the Pixel Tablet are both new and exciting devices that offer a unique experience. The Pixel 7A is a great option for budget-minded consumers.

Conclusion

Google I/O 2023 was a major event for the tech industry, and it showcased Google's commitment to innovation. The announcements at I/O 2023 will have a significant impact on the way we interact with computers and the way we access information.

The new hardware products, software features, and cloud computing services announced at I/O 2023 have the potential to make our lives easier, more efficient, and more creative. They also have the potential to revolutionize the way we do business and the way we learn.

Google is a company that is constantly pushing the boundaries of what is possible. With its continued investment in innovation, Google is sure to continue to shape the future of technology.

In this blog post, we will discuss the concept of prompt engineering and how it can be used in AI applications. Prompt engineering is a technique that allows AI systems to automatically generate descriptions of tasks they are asked to perform. This can be useful for tasks such as understanding natural language or performing automated reasoning.

Prompts can come in many forms, including questions posed by humans or data sets provided by other machines. In general, prompts should be easy for the AI system to understand and answer correctly. Additionally, prompts should provide enough information so that the AI system knows what task it needs to complete in order to satisfy the user's request.

Prompt engineering is a relatively new field of research, but it has already been used in several applications. For example, Google's search engine uses prompts to help users find information on the web. In this case, the prompt is a question posed by the user and the search engine's response is a list of relevant web pages. Another example of prompt engineering is when you use ChatGPT to find answers to your questions. In this case, the prompt is a question posed by the user and ChatGPT's response is an answer to that question.

Large Language Models (LLM)

Large language models are a type of machine learning model that is used to learn how to generate text from data. They are often used for tasks such as predicting the next word in a sentence or understanding the meaning of a document.

A large language model can be thought of as an artificial neural network that is designed to handle large amounts of data. This allows it to learn how to generate text from scratch, rather than just using pre-existing examples.

This type of model has been shown to be more effective than traditional machine learning models when it comes to dealing with complex texts and languages.

Types of LLM

1. Base LLM: Base LLM does text continuation based on its knowledge of the language and the knowledge base provided. Examples of Base LLM include auto-completion.

2. Instruction-Tuned LLM: Instruction-Tuned LLM reads the prompt, understands it, and gives an appropriate response. An example of an Instruction-Tuned LLM is ChatGPT.

Principles of Prompt Engineering

Be clear and specific.

When it comes to prompt engineering, it's crucial to ensure that the prompts used are specific and tailored to the task at hand. Specific prompts provide clear direction and guidance to the model, increasing the chances of generating high-quality responses.

To illustrate, a specific prompt for a language translation task might be: "Translate the following sentence from English to Spanish: 'The quick brown fox jumped over the lazy dog.'" This prompt provides the model with a clear objective and specific input to work with, resulting in a more accurate translation.

On the other hand, a non-specific prompt for the same task might be: "Translate some English sentences to Spanish." This prompt is vague and lacks direction, which can lead to a lower-quality output from the model.

In summary, specific prompts are essential in prompt engineering as they provide a clear objective and input for the model to work with, resulting in higher-quality output. Non-specific prompts can lead to ambiguity and lower-quality output.

Ask to answer in a specific format

Asking for the output in a specific format is a great way to ensure that the LLM's response is compatible with the systems and software being used. Here are some examples of specific prompts that request the output in different formats:

1. JSON: "Provide a JSON object with the following fields: 'name', 'age', and 'email', and their respective values." This prompt would instruct the LLM to generate a JSON object that includes the specified fields and values.

2. XML: "Create an XML document that includes a root element called 'books' and child elements for 'title', 'author', and 'published_date'." This prompt would direct the LLM to generate an XML document that follows the specified structure and contains the specified elements.

3. HTML: "Generate an HTML table that displays the following data: 'Product Name', 'Price', and 'In Stock'." This prompt would instruct the LLM to create an HTML table that displays the specified data, likely for use on a website or in an email.

By requesting specific formats like these, you can ensure that the LLM's response is easily integrated with other systems or software, saving time and effort in post-processing.

Give context or additional information

Providing context or information about the question being asked is a helpful way to improve the accuracy of the LLM's response. Here are some examples of how context or information about the question can be provided:

1. Context: "In the context of a financial report, can you provide the revenue figures for the last quarter?" By specifying the context of the question, the LLM can better understand the purpose and scope of the request, leading to a more accurate response.

2. Information about the question: "Can you provide me with the definition of 'epistemology'?" By providing information about the kind of question being asked, the LLM can better understand the type of information being sought, and provide a more relevant and accurate response.

3. Clarification: "I'm looking for a hotel in New York City that is within walking distance of Times Square. Can you suggest any options?" By providing clarification about the request, the LLM can better understand the specific criteria being used to evaluate options, leading to a more accurate and relevant response. In summary, providing context or information about the question being asked is a useful way to improve the accuracy of the LLM's response, by helping the model understand the purpose, scope, and criteria of the request.

Reframe the question (prompt)

If the LLM is not giving the desired output, providing additional information or reframing the question can often help improve the quality of the response. Here are some examples of how this can be done:

1. Providing additional information: "Can you recommend a good restaurant in the downtown area of San Francisco?" If the LLM's initial response is not helpful, providing additional information such as the type of cuisine or price range you are looking for can help the model better understand your preferences and provide a more relevant recommendation.

2. Reframing the question: "What is the best way to approach a difficult conversation with a colleague?" If the LLM's initial response is not helpful, reframing the question to provide more specific or actionable information can help the model provide a more useful response.

3. Asking follow-up questions: "Can you explain more about what you mean by 'successful marketing campaign'?" If the LLM's initial response is vague or unclear, asking follow-up questions can help clarify the request and improve the quality of the response.

By providing additional information, reframing the question, or asking follow-up questions, you can help the LLM better understand your needs and provide a more accurate and relevant response.

Limitation

One of the known limitations of language models, including LLMs, is that they may generate responses that contain inaccuracies or make up information if the question asked is beyond their knowledge base or the data they were trained on. This is because LLMs rely on statistical patterns in the data they have been trained on to generate responses, and if the input is outside their training data, they may not have enough information to provide a meaningful or accurate response.

Furthermore, all language models, including LLMs, are limited by the quality and quantity of data they were trained on. If the training data is biased, incomplete, or inaccurate, the LLM's responses may also be biased, incomplete, or inaccurate. Additionally, if the training data is limited in scope, the LLM may not be able to generate responses that are relevant or useful for a wide range of tasks or applications.

To address these limitations, it's important to carefully evaluate the accuracy and relevance of the LLM's responses and to continue to train and refine the model using high-quality data that reflects a diverse range of perspectives and use cases. By doing so, we can continue to improve the accuracy and usefulness of LLMs and other language models.

Conclusion

In conclusion, Language Models, especially the recently popular LLMs, have the potential to revolutionise the way we interact with natural language. However, like any other technology, they also have their limitations. To make the most of these models, it's important to understand their strengths and limitations and to use them in conjunction with other tools and methods to achieve optimal results.

Some ways to optimize the use of LLMs include providing specific and relevant prompts, providing context or information about the question or task, and reframing the question or asking follow-up questions if the initial response is not helpful. Additionally, it's important to be aware of the limitations of these models, such as their tendency to generate inaccurate responses if the input is beyond their knowledge base, or if the training data is biased or incomplete.

To overcome these limitations, we need to continue to train and refine LLMs using high-quality data that reflects a diverse range of perspectives and use cases. With careful use and continued refinement, LLMs have the potential to improve many areas of our lives, including communication, education, and research.

But coding directly in WebGL is nothing less than doing self-harm. That's where Three JS comes into play. This Javascript library takes care of all the painful elements of WebGL and provides an abstracted syntactic sugar for us to work with. We will see in later posts, how we can make custom functions in WebGL and append that to Three js to extend its functionalities.

Before we begin, take a look at some of these amazing websites built using Three.js

1. Github Homepage
2. Bruno Simon Portfolio
3. Lusion Co
4. Particle Love by Edan Kwan

You can see a lot more of these real world usage of Three js over at Three JS Website

Initial Setup

There are three ways to embed Three.js to your website. (Coincidence)

Option 1 - Cloning Three js Github Repository

Before doing this, I recommend you to create a folder structure for all your projects. Otherwise, it will be hard for you to find something in the future. Once that is done. Simply clone the repository to your desired folder.

git clone https://github.com/mrdoob/three.js.git three

Option 2 - Three js CDN

This would be the easiest way to start, just embed this script to your webpage

<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r126/three.min.js" integrity="sha512-n8IpKWzDnBOcBhRlHirMZOUvEq2bLRMuJGjuVqbzUJwtTsgwOgK5aS0c1JA647XWYfqvXve8k3PtZdzpipFjgg==" crossorigin="anonymous"></script>

But as you will see later, the core Three js library only include simple shapes, textures and materials. If you want to add an external model or do something that is not included in this js file then you need to import those specific files too. And in a real world project, by using this method you will end up importing way too much files that you won't even fully utilize.

Option 3 - Three js Project Starter Kit

My Three js sensei, Bruno Simon, provided us a starter pack that uses webpack to assemble all packages. I took that as base and created a three js starter kit for myself so that I don't have to write the boilerplate code everytime. You can use this kit by using the following git command.

git clone https://github.com/Idiomatic-Programmers/threejs_starter_kit.git <YOUR_PROJECT_NAME>

Don't forget to replace with something meaningful to your project.

If you haven't already, you need to install Node.js and NPM for this to work. Checkout Node JS Documentation for more information.

Setting up your first project

For demonstration purpose, I am going to name our project "Hello World"

git clone https://github.com/Idiomatic-Programmers/threejs_starter_kit.git hello_world

This will create a folder called hello_world inside which you will find all the starter code already written.

FYI - If you want to learn and master Three js and can afford to buy a $95 course, checkout Bruno Simon's Three JS Journey, it pretty great, very in-depth, and easy to follow along. Plus, you get access to a private Discord server where you can meet thousands of amazing developers and you can clear your doubts directly from Bruno. But if you can't afford it, then follow along. I will try my best to share the knowledge I gained from this course.

Open the hello_world folder in your favourite code editor and navigate to script.js file located in src folder. Now select everything and just delete it because we will be coding just that.

Before we continue, don't forget to install the packages we need by doing.

npm install

The beauty of webpack is that we can do most of our things using Javascript without ever touching the HTML file. I'm assuming you have deleted everything from script.js file.

So in the very first line, import our css file

import './style.css'

Then obviously we need to import our three js library.

import * as THREE from 'three'

Before we write our next line of JS, let's understand the HTML structure of our project. If you open index.html in the src folder, you will find there's standard HTML boilerplate with a canvas element in the body. This canvas will render everything we do in three js, which means we must have a reference to this canvas element in out js file.

const webGLCanvas = document.getElementById('webgl')

With these three lines in our js file, we can officially start writing our three js code. But first, let's learn some theory.

Fundamental Elements in Three JS

Every three js project will contain these three elements.

a. Scene

In the genre of Film Making, a scene is a place or setting where the action takes place. A scene holds everything a movie needs. If you take an example from this amazing Steven Spielberg movie Catch me if you can.

This particular scene contains everything that is required to move the story forward.

Much like that, a scene in Three js will hold everything that is required to tell our story, cars, trains, builings, or even wierd 3D figures, everything that you need to show will be included in a scene.

b. Camera

Again taking the example of the aforementioned movie, a camera will only show a particular section of a scene that is required to tell the story.

In Three JS, there are two main types of camera,

1. Perspective Camera

   A Perspective camera in three js is a Frustum Shape (3D Trapezoid) and the user will only be able to see what's inside this frustum. You can adjust the size of this Frustum to get the best results. And everything in the shape is Perspective Projected meaning farther objects appear smaller than nearer objects.

   This camera simulates real world so we will be using this camera a lot in coming posts.

   

2. Orthographic Camera

   An Orthographic camera in Three js is a Box instead of a Frusturm like in Perspective Camera and it uses an Orthogonal Projection which means the size of the object remains same regardless of their distance from the camera. These type of cameras are useful to design maps of the environment.

   

There are other type of cameras in three js but we won't be using them. But if you would still like to learn about them here's a list of all cameras linked to their documentation.

1. ArrayCamera
2. CubeCamera
3. OrthographicCamera
4. PerspectiveCamera
5. StereoCamera

c. Renderer

A renderer is an element that displays whatever the camera sees onto a canvas. It is a screen on which the 3D scene is projected. And that is the only thing we need to know about this.

Create your first 3D environment

Coming back to script.js file, we have three lines which imports everything we need.

Next we will proceed to define the three fundamental elements in three js.

const scene = new THREE.Scene()

This will initiate an empty scene on our project, next we need a camera to see what's in a scene. We will be using Perspective Camera which takes in four main parameters.

1. Field of View (fov): This specifies how wide your camera can see, in degrees.
2. Aspect Ratio (aspect): As the name suggests, this specifies the aspect ratio of your scene. is it 16:9, 16:10, 4:3 etc. If you would like to find out more about aspect ratios, check this article
3. Near Distance (near) and Far Distance (far): These two parameters will specify the boundaries of the Frustum. You can only see those objects that are between the near and far values of the camera.

const sizes = {
    width: window.innerWidth,
    height: window.innerHeight
}

We will define an object that will contain the height and width of the renderer. Everything the camera sees in 3D will be projected to this 2D plane. In our case, the renderer will take up our entire screen.

const camera = new THREE.PerspectiveCamera(75, sizes.width / sizes.height, 0.1, 1000)

Notice for aspect ratio, we have set the ratio between renderer width and height. This is because we have no control over the aspect ratio of user device. If you are 100% sure that only people with devices that has a 16 by 9 ratio, then you simply write 16 / 9 instead of sizes.width / sizes.height and you can experiment with the near and far values that work best for you. You can copy what I have written it doesn't matter, atleast for what we are doing.

Finally, we need a renderer. Three js provides many renderers, but we are only interested in one of them called WebGLRenderer which takes an object as a parameter that has many attributes that you can look at in the documentation

For now, we will be passing one attribute called canvas which you guessed it, takes the reference to our canvas.

const renderer = new THREE.WebGLRenderer({
    canvas: webGLCanvas
})

We also need to set the size of this renderer.

renderer.setSize(sizes.width, sizes.height)

Alright, that's it. Let run our code. Run this code in your terminal

npm run dev

Wait, what? We only see a blank black screen? Let me check it there is any error in Inspect Element Console.

And my right click isn't working? What's happening?

I lied, we need to do a bit more work.

We need to add our camera into the scene.

scene.add(camera)

and actually render the scene. You can do this by calling the render method of renderer and pass the scene and camera into it.

renderer.render(scene, camera)

Now save and refresh the page. Still a black screen? It's completely normal. Remember the scene currently contains a camera and we can only see what the camera see and there's nothing else to see.

If you don't like this black screen then you can change the background colour of the scene like this.

scene.background = new THREE.Color("#f7f7f7")

You cannot directly pass the hex code, you need to wrap it around a THREE.Color object.

If you refresh your page now, you can see the background has been changed to nice light greyish colour.

Now let's create a Sphere.

You need three things to create any object in Three js.

1. Geometry: This holds a general shape of the object
2. Material: This hold information about the texture, color, opacity, etc.

3. Mesh: This combines the Geometry and Material into an object.

   Three js provides Geometry for almost all basic shapes like sphere, box, torus, etc. For making a sphere, there is something called SphereBufferGeometry that takes three parameters.

4. Radius: The radius of the sphere

5. Width Segments
6. Height Segments

Don't worry about Width and Height segments now, I will show you once we built our Sphere.

const sphereGeometry = new THREE.SphereBufferGeometry(1, 32, 32)

We have a radius of 1 units and 32 Width and Height segments.

In three JS, it'd better to decide what a unit is. For me 1 unit is 1 meter.

Then we need a Material, we will see different type of materials provide by Three JS and also you can create your own material using a laguage called GLSL (OpenGL Shading Language). But for now, we'll use MeshStandardMaterial

const sphereMaterial = new THREE.MeshStandardMaterial()
sphereMaterial.color = new THREE.Color("#ffffff")

We can set the colour of this material by updating the color attribute. Also, remember we need to pass color as a THREE.Color object.

Finally, we will create a mesh using THREE.Mesh and pass in our geometry and material.

const sphere = new THREE.Mesh(sphereGeometry, sphereMaterial)

And don't forget to add the sphere to scene.

scene.add(sphere)

One last thing, you should always render the scene at the end of the script, so move this line to the end.

renderer.render(scene, camera)

Nothing? Don't worry. Right now your camera is inside the Sphere.

Three js calulates the distance between a vertex and the camera, if that distance is less that near or far parameters of the camera then that object will not be shown. In this case, the distance between Sphere and Camera is 0 and since it is smaller than the near parameter which is 0.1, the sphere will not be shown.

We can move back our camera by moving it few units towards positive z direction.

camera.position.z = 5

Now you can see a Black circle. But why is it black? We did set the color as White (#ffffff).

Some of you might have guessed it. We have no light

Three js actually has this amazing method called letThereBeLight(), which automatically adds all the light we want.

Just Kidding, life isn't that simple, let us actually add some light to our scene.

We will see all kind of light throughout this series, but for now we will look at a light called Directional Light, which gives parallel rays of light just like our Sun.

The DirectionalLight takes two parameters, the color of the light, which will be a Hex code and the intensity of the light.

const directionalLight = new THREE.DirectionalLight("#ffffff", 0.5)
scene.add(directionalLight)

Always remember to add your objects to scene.

We can see some light at the top, but actually the light is inside the sphere. Let's move it using position method of the object.

For any object in three js, you can move it using one of two ways

1. Changing the x, y, z positions one by one

directionalLight.position.x = 30.25
directionalLight.position.y = 30
directionalLight.position.z = 30

2. Using set() method:

directionalLight.position.set(30.25, 30, 30)

You can use whatever method, which you feel is right.

This is better. But there is very harsh shadow. Let's introduce a new kind of light to fix this issue called Ambient Light.

Ambient Light adds the same amount of light everywhere in the scene, and it does not cast any shadow. Using AmbientLight is similar to Directional light, the only difference is that you don't have to move it around.

const ambientLight = new THREE.AmbientLight("#ffffff", 0.5)
scene.add(ambientLight)

And there it is, a beautiful Sphere suspended in our scene.

Resizing

If we resize the window, you'll notice that the canvas is not changing it's size. We need to do bit of work to fix that.

window.addEventListener('resize', () =>
{
    // Update sizes
    sizes.width = window.innerWidth
    sizes.height = window.innerHeight

    // Update camera
    camera.aspect = sizes.width / sizes.height
    camera.updateProjectionMatrix()

    // Update renderer
    renderer.setSize(sizes.width, sizes.height)
})

We have created a resize event listener and just updated the aspect ratio and renderer size. We also need to tell the camera to recalculate its projections, which is done by the updateProjectionMatrix() method.

Animation

It wouldn't be fun if we don't animate our sphere. Let's create a function called animate() and inside that call the renderer.render method. Make sure that you are writing this code at the end of your script.

const animate = () => {
    renderer.render(scene, camera)
}

animate()

This is not enough, We need to tell our browser that we wish to animate our scene. Which can be done by calling the window.requestAnimationFrame method and pass it reference to our animate function.

const animate = () => {
    window.requestAnimationFrame(animate)
    renderer.render(scene, camera)
}

animate()

Animating our scene isn't that simple as calling requestAnimationFrame. We need to figure our some math for that.

For our scene, I'm thinking to animate Sphere up and down in a smooth motion. This can be done by using the sine function in trigonometry. But before that, we need some way to tell the passage of time. Conveniently, Three js provides a class for that called Clock.

const clock = new THREE.Clock()

We can get the elapsed time by calling the getElapsedTime method of the clock object.

Write this code in our animate function.

const elapsedTime = clock.getElapsedTime()

sphere.position.y = Math.sin(elapsedTime * 150 * Math.PI / 180) * 0.125

Save this and view the page, your sphere will be moving Up and Down smoothly.

You can play with the math and change the axis, and see what happens.

Before we leave, take a look at what we will be building throughout this series.

You can check out the live demo website

That is all for this post, in the next post we will explore some few more shapes, materials, cameras and lights. If you have any query be sure to comment below, we will get back at you as soon as possible,

Pandas is an essential tool used by Python developers used for data analysis purposes, but what is the point of the analysis if we are not able to provide that insight to the end user. When Django and Pandas are used in conjunction we can create applications that are not only developed faster, but are also smart in utilizing the data that is collected from the user. I recently stumbled on one such use case, where the user interacts with the frontend and turns some filters and then the backend processes that data, and then provide the aggregated data as an Excel or CSV file to the user. I will be describing how I went about doing that below.

Prerequisites

In order to follow along this article, I am assuming you already have a Django project that is using Pandas library ready.

1. Django Documentation
2. Pandas Documentation

Requirements

In addition to Django and Pandas project ready, you would need to install Openpyxl.

Openpyxl

Openpyxl is a tool that allows to read and write Open Office XML formats such as Excel 2010 xlsx/xlsm/xltx/xltm files.

pip install openpyxl

Once installed, you can create Excel files natively from Python using a simple code like this.

from openpyxl import Workbook

workbook = Workbook()
sheet = workbook.active

sheet["A1"] = "ID"
sheet["B1"] = "Key"

sheet["A2"] = 1
sheet["B2"] = "Test Key"

workbook.save(filename="test.xlsx")

This will create a file called "test.xlsx". Try it out yourself.

HTTP Response

As you know, Django uses HTTP Request and Response to communicate with the client. Therefore all data must be converted to Byte String before sending it over to the client.

A Typical HTTP Response looks something like this.

For more information, checkout this article by Mozilla about HTTP Messages.

As you can see in the above picture, every HTTP response has a body section which is a Byte Array and a Header called Content-Type specifies what type of content it is and which character set the browser should use in order to decode the bytes. In the above case, the server is sending a text/html data and the browser need to use iso-8859-1 charset.

In our case, we are sending bytes that are related to Excel spreadsheets. So googling for a bit I found the content-type header for that is this. Source: Mozilla article about Common MIME Types

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

What are MIME Types?

MIME is short for Multipurpose Internet Mail Extensions which is a standard that indicates the nature and format of a document sent via the Internet.

Important: Browsers use the MIME type, not the file extension, to determine how to process a URL, so it's important that web servers send the correct MIME type in the response's Content-Type header. If this is not correctly configured, browsers are likely to misinterpret the contents of files and sites will not work correctly, and downloaded files may be mishandled.

For more information, checkout MDN Article about MIME Types

The structure of a MIME Type is according to IANA (Internet Assigned Numbers Authority) is like this.

type/subtype

Therefore, in the MIME Type for Excel files, the type is "application", which belongs to any kind of binary data that cannot be directly decoded to a human-readable form (text, HTML, etc.). Files with type application require some sort of external or third party software to decode and read such as PDFs, Zip files, Excel files, etc.

Download Excel Files using Django and Pandas

Finally, we will see how we can send the Pandas dataframe to client as an excel file.

I am assuming that you already have the code for pandas so I will do some abstraction in a variable called data.

First we will import to the inbuilt library, BytesIO so that we can write the excel file as a Byte array.

BytesIO is a library using which we can write data onto system memory instead of writing them as a file.

from io import BytesIO

Then we will use Python context manager to open a Byte buffer on which we can write the excel file. Context Managers allow you to allocate and release memory precisely when you want it.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
        # You can add multiple Dataframes to an excel file
        # Using the sheet_name attribute
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

    filename = "analytics_data.xlsx"

    # imported from django.http
    res = HttpResponse(
        b.getvalue(), # Gives the Byte string of the Byte Buffer object
        content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
    )
    res['Content-Disposition'] = f'attachment; filename={filename}'
    return res

We will be using context managers to open and close files because that's more efficient than manually closing and releasing the memory after use. If you already know how to export a set of data frames as an excel file, that code should look like this.

with pd.ExcelWriter('data.xlsx') as writer:
    data1.to_excel(writer, sheet_name="DATA 1", index=False)
    data2.to_excel(writer, sheet_name="DATA 2", index=False)

We will simply take this code and wrap it around a BytesIO context manager which gives us a memory buffer b as a file to work with. We will simply take that memory buffer and pass it to pd.ExcelWriter() class and the rest of the code will be the same this that context manager.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

Now that we have our Excel data written in a memory buffer, we can simply convert that to ByteArray using the method b.getvalue() which we will pass as a parameter in Django HttpResponse with appropriate content_type that we discussed earlier.

filename = "analytics_data.xlsx"

# imported from django.http
res = HttpResponse(
  b.getvalue(), # Gives the Byte string of the Byte Buffer object
  content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)

In addition to the Content-Type header, we also need to provide another header called Content-Disposition that tells the browser if we want to show the data in the browser download and save it as a local file as an attachment. We can also pass a filename with this header.

res['Content-Disposition'] = f'attachment; filename={filename}'
return res

Here is the complete code snippet for Django view to download the excel file.

def download_analytics_endpoint(request):
    with BytesIO() as b:
        data = get_analytics_data()

        with pd.ExcelWriter(b) as writer:
            data.to_excel(writer, sheet_name="Data", index=False)

        filename = f"analytics_data.xlsx"
        res = HttpResponse(
            b.getvalue(),
            content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
        )
        res['Content-Disposition'] = f'attachment; filename={filename}'
        return res

That is all, now you just have to create a Django URL for this function and you can download any file you want, just convert that file to a Byte string and use the correct Content-Type header.

The diagram below shows the basic architecture of a GAN.

Source: https://developers.google.com/

Generative Network takes some random noise and outputs some random noise. This output noise is passed to a discriminator along with the real image or data as a ground truth based on that both Discriminator and Generator is trained.

As you can see, the concept of GAN is very simple. When I started learning about GANs I thought I can easily implement one of those. But boy I was wrong. In reality training a GAN is extremely hard both Generator and Discriminator must be trained side by side if one overpowers the other it won't work, we will talk about all the problems you might face while training a GAN. But now let's look at a simple GAN in Pytorch. We will be using the MNIST Dataset[^1] for this post.

Understanding the Dataset

The MNIST dataset is a huge database of handwritten numbers from 0 to 9 used for Optical Character Recognition or reading numbers from an image.

This dataset consists of 28x28 images of handwritten numbers where each pixel contains either a zero or a one.

The computer vision extension of PyTorch, Torchvision, provides this dataset which we can download using the following code snippet.

mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)

But before we can run this code, we need to import some libraries.

import torchvision.datasets as datasets
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

You can install the torchvision library using this pip command.

pip install torchvision

The DataLoader class is used to load the data to memory in batches, this prevents your system from running out of memory while training.

Transforms class is used to make random augmentations to the image such as random rotation, resize, crop, etc. but for now we will only normalise the images to range from -1 to 1.

The entire function that you can copy paste is this.

def load_dataset():
    transformations = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, ), (0.5, ))
    ])
    mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)
    return DataLoader(mnist, batch_size=32, shuffle=True)

Implementing the Generator

In this section, we will implement a generator network which will take a random noise vector of size 100 and convert it into a vector of size 784 which we will then convert to 28x28

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.gen = nn.Sequential(
            nn.Linear(100, 256),
            nn.LeakyReLU(0.01),
            nn.Linear(256, 784),
            nn.Tanh(),  # make outputs [-1, 1]
        )

    def forward(self, x):
        return self.gen(x)

We will be implementing the original GAN created by Ian Goodfellow in this paper [^2]

Implemeting the Discriminator

Next, we will implement a discriminator that will take the vector of size 784 that may be generated or from a real image.

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.disc = nn.Sequential(
            nn.Linear(784, 128),
            nn.LeakyReLU(0.01),
            nn.Linear(128, 1),
            nn.Sigmoid(), # make outputs [0, 1]
        )

    def forward(self, x):
        return self.disc(x)

Hyperparameters

As I mentioned earlier, GANs are extremely hard to train, one of the reasons is that a GAN is very sensitive to the initial values or hyperparameters. You have to follow the original papers to get the training right. Otherwise, you might have to spend many days optimizing the hyperparameters.

config = {
    'device': "cuda" if torch.cuda.is_available() else "cpu",
    'lr': 3e-4,
    'epochs': 50,
        'batch_size': 32
}

In our case, we will be using the same parameters as it was said in the paper, that is, a learning rate of 0.0003 and 50 epochs.

What is a learning rate?

Simple answer, it's the rate at which a machine learning model learns. Smaller the number, the slower it learns and higher the number the faster it learns. For more info, check out our post about Perceptrons.

Training Time

Finally, it's time to actually generate some handwritten numbers, that is, train our GAN. First let's create the objects for Generator, Discriminator, and their optimizers, we will be using the Adam optimizers.

ADAM or ADAptive Moment optimiser is an algorithm[^3] used to update the weights such that the overall error goes down.

disc = Discriminator().to(config['device'])
gen = Generator().to(config['device'])

optimiser_g = optim.Adam(params=gen.parameters(), lr=config['lr'])
optimiser_d = optim.Adam(params=disc.parameters(), lr=config['lr'])

We also need to define a loss function before we train, we will be using the Binary Cross Entropy loss[^4] function to calculate the error of our model.

Binary Cross Entropy (BCE) Loss

This loss formula is used to calculate the distance between two probability distributions.

$BCE = ℓ(x,y)=L=(l_1​,…,l_N​), l_n​=−w_n​[y_n​⋅logx_n​+(1−y_n​)⋅log(1−x_n​)]$

loss_fn = nn.BCELoss()

Now we will write our training step which is one cycle of generating and discriminating a handwritten number compare with an original number and update the models.

for epoch in range(config['epoch']):
    for batch_idx, (real, label) in enumerate(train_data):

        noise = torch.randn(config['batch_size'], 100).to(config['device']) # Create a random probability distribution
        fake = gen(noise) # Generate a fake number
        disc_real = disc(real).view(-1) # pass the real number through the discriminator
        lossD_real = loss_fn(disc_real, torch.ones_like(disc_real)) # calculate the loss for real image
        disc_fake = disc(fake).view(-1) # pass the fake number through the discriminator
        lossD_fake = loss_fn(disc_fake, torch.zeros_like(disc_fake)) # calculate the loss for fake image
        lossD = (lossD_real + lossD_fake) / 2 # calculate the average loss

        # update the weights for the discriminator
        disc.zero_grad()
        lossD.backward(retain_graph=True)
        optimiser_d.step()

        output = disc(fake).view(-1)
        lossG = loss_fn(output, torch.ones_like(output)) # calculate the error between the fake image and the true image

        # update the weights for the generator
        gen.zero_grad()
        lossG.backward()
        optimiser_g.step()

This code is heavily inspired by a youtube video by Aladdin Persson.

Find the complete code at our Github Repo

If you have any questions, feel free to comment below.

Footnotes

[^1]: MNIST Database - “THE MNIST DATABASE.” MNIST Handwritten Digit Database, Yann LeCun, Corinna Cortes and Chris Burges, yann.lecun.com/exdb/mnist/. [^2]: Paper by Ian Goodfellow - Goodfellow , Ian J., et al. “Generative Adversarial Networks.” ArXiv.org, 10 June 2014, arxiv.org/abs/1406.2661v1. [^3]: Adam: A Method for Stochastic Optimization - Kingma, Diederik P., and Jimmy Ba. “Adam: A Method for Stochastic Optimization.” ArXiv.org, 30 Jan. 2017, arxiv.org/abs/1412.6980. [^4]: Binary Cross-Entropy Loss - Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and All Those Confusing Names, gombru.github.io/2018/05/23/cross_entropy_loss/

Social Networking Websites are being asked to make their moderation policies more stringent and expressing your opinions freely on the internet without any repercussions will soon become next to impossible. Even now, anyone can take down a post or video. Talking from a developer's perspective, you wouldn't ever think about an open source project being taken down, but we all saw recently that youtube-dl got taken down because of a DMCA notice.

We get our internet connection at the merci of our Government even though we as a consumer pay a premium to use this luxury that is now a necessity, the government has the ultimate power to block and ban anything from the internet at least in their country.

You all know that China heavily censors their internet to protect their communist beliefs and to shut anyone who criticises the president and the government. Ok but that's China.

It's not just China. India for example, the largest democracy in the world, banned more than 150 apps and websites including PubG, Alibaba and Ali Express just because they were created by Chinese companies. You might say, that's politics.

It's not just politics, a few years ago India banned all Adult content from the Internet just because some politician didn't like that. A massive corporate entity persuaded the government to ban two music videos just because they didn't like the criticism in the lyrics.

How the government can block a website or content?

There are several ways a government entity can ban/block a website, app or any content on the internet.

1. Issuing a court order - The government can ask the judicial system to issue a court order to the company that owns the website or content and the company have to remove such content as long as they are operating in the country.

2. DNS Lookup Blocks - I will deep dive into this topic later in the post, but in simple terms, the government can ask the DNS providers to not resolve requests for a set of domains if the request is coming from their country. This has loopholes like the DNS Provider can decline the request if the company that owns the DNS Servers does not belongs to the country. These types of blocks can be bypassed by using a custom DNS like 1.1.1.1

3. ISP Blocks - The Internet Service Provider has the ability to not serve certain requests that are coming from or going to a banned IP address. I will talk about this later in the post.

But for all these bans can be bypassed by using a VPN.

There are some countries, where using VPN is banned and the ISP and the government can track and fine VPN users even though they are using an end to end encrypted network.

I happened to live in one of these countries, where using VPN is not illegal but using a VPN to bypass blocked content is illegal. (the only thing I need a VPM for)

This is when my curiosity peeked. How is my ISP able to detect that I am using a VPN and flag accordingly even though I'm using an end to end encrypted connection that too over HTTPS?

Bandwidth Throttling

You might have noticed in the title, I've written "throttles" in parenthesis instead of just blocks. That is because, in this country's laws, using VPN is not illegal. But using a VPN to bypass blocked websites and content is (basically only reason, I use VPN for)

Because of this reason, the ISP never completely blocks a VPN but throttles it. Now what does that mean?

Bandwidth throttling is the intentional slowing or speeding of an internet service by an Internet service provider (ISP)

This is exactly what I was experiencing. If you're not living in any of the European Union, your ISP most likely has two or even more that two network channels, a fast channel where you get very fast internet speed like 4G or 5G, and a slow channel where you get only a fraction of the internet speed you are paying for.

In my research, I speculate that my particular ISP has a third channel where there is next to nothing speed maybe like 20 or 30 bytes per second. And whenever they detect that I am using a VPN, they will flag that and put my connections on the third lane.

How a normal internet connection works?

Let's say that you want to connect to some website, idiomaticprogrammers.com, you will type idiomaticprogrammers.com onto your browser and it parses and detects that it is a website url.

Then the browser creates a packet that in simple terms looks like this.

This application, let's say Chrome, wants to get the webpage in idiomaticprogrammers.com and send this packet to the router. There are many layers to this process which is not relevant in this post I will talk about them in detail in a future post.

You router looks at this packet and adds a record to its entry and replaces the local IP address with the router's IP which is actually connected to the internet.

After that, the router searches for the IP Address to where you want to send this packet to. But only information we've provided is the domain name idiomaticprogrammers.com, so the router first prepares a different packet which looks something like this.

This Router want to do a DNS lookup for idiomaticprogrammers.com and send it to a Domain Name Server. Which usually is your ISP but your can manually set the IP Address of the DNS in your router like 1.1.1.1 is Cloudflare's DNS and 8.8.8.8 is Google's DNS (I personally use 1.1.1.1)

What is DNS Lookup? It is a process of asking what is the IP address associated by this domain name.

Once the router knows the IP address for the domain name it simply replaces the domain name with the IP address and sends the request to the ISP.

Then the ISP looks at the packet can see that you want to go to this IP so it sends the request to that IP and then returns whatever information the web server provides back to the client.

Note that, if a website is using HTTPS, that is you can see a secured lock in front of the URL, there is no way for the ISP to know what information is returned by the website. Everything is encrypted end to end. I will be making an entire post about how this works in the future. But for now, this is a basic idea of how things work on the internet.

How ISP blocks a particular domain?

Remember how a router sends a DNS Lookup request to the Domain Name Server? No matter what DNS you are using either you ISP or some custom one. This request has to pass via your ISP because they are the ones who controls your internet and responsible for movement of data to and from your computer.

Every ISP has a long list domain provided by the government that they decided to block. Whenever you send a DNS Lookup request, the ISP checks if this domain is in that list. If it is, the it will be flagged and you will see a message like this domain does not exist or a page by the ISP saying this domain is blocked in this country.

Now you may ask, what if I entered the IP address directly? That might be possible, provided two conditions, 1. This IP address is not in the blacklist of the ISP and 2. The web server is configured to receive requests from the bare IP address. Believe it or not, we backend engineers do a lot of heavy lifting for you frontend devs.

How VPN works?

Going forward with the same example, let's say (god forbid) idiomaticprogrammers.com is banned in your country but you want to read that latest article we've posted. You connect to a VPN paid or free doesn't matter, at a network level the following is happening.

Your local IP is let's say 192.168.1.27 and somewhere on the planet there exist a server that has it's own private IP let's say 10.0.0.2.

Note the word private, meaning you have no way to connect to the server with private IP 10.0.0.1. What VPN does is this.

As you can see in my diagram, the VPN treats your computer as if it's a part of their network hence the word "Virtual" Private Network.

Now if you connect to idiomaticprogrammers.com your computer will make a packet like this, compare this with the packet the computer creates when it's not connected to any VPN.

There are extra information and this time it's going straight to the public IP address of the VPN, The VPN software you're. using crypts this packet and makes a new packet which essentially says, 192.168.1.27 wants to send this encrypted message to this public address let's say 123.66.6.234. The router will make this entry and forward this to the ISP

ISP looks at the packet, and just forward it this IP and it does as instructed.

The VPN server looks at the packet, decrypts it and because there are protocols, forwards it to 10.0.0.2 which then does all other processes like DNS Lookup and GET request to the website, encrypts whatever data it gets and simply forwards them back to us.

Now for idiomaticprogrammers.com, that VPN server makes all the requests and no one but the VPN and ISP knows our identity and that is what these VPN companies say "Anonymous", I say not really.

How ISP throttles VPN?

In the last part you've seen that those encrypted VPN connections pass through ISP. So theoretically, an ISP and make a list of all the VPN servers and whenever some data comes from these Public IP addresses it just says, "You might be a VPN, we have this special channel for you sir" and simply place our connection in a low bandwidth channel where the Internet speed is barely 20 or 30 bytes per second.

The same happens when you connect via TOR Protocol.

If you find some inconsistencies or theoretical error in this post please leave a comment below and I will correct myself as soon as possible.

Data Preprocessing

Acquire Raw Data

Extract knowledge from Data

Prepare the data for the Neural Network

Build your first Neural Network

Training and Validation

Finalise the pipeline

Nuxt.js is a framework for building Vue.js apps, Nuxt basically gives you everything out of the box that Vue does not. Features such as faster page loads, SEO Optimisation, Standardised folder structure, Modular architecture, etc.

You can find a plugin for almost anything you want to add in your site. You can convert your application into a PWA in just 1 line of code.

In addition to that, Nuxt provides you with the feature to make your site completely static meaning you can prerender all the HTML, CSS and JS files required to run your site/application. And nuxt has relatively fasted build time compared to Angular or React and has a pretty smooth learning curve.

Installation

Nuxt provides a simple npx module to create the initial folder structure. Just write the following on a terminal or command prompt

npx create-nuxt-app MyBlog

Here "MyBlog" is the name of our blog that we are going to build. Feel free to be creative and name something amazing for your blog.

This step will ask you questions such as Project Name, If you will be using Javascript or Typescript (I am going to use Javascript), Will you use NPM or Yarn to install dependencies (I'll use NPM), What CSS framework you will be using (I'm going to import Bootstrap using CDN because that's easier for me, so I selected None)

After selecting the UI (CSS) Framework, it will ask you which Plugins to include. I'm going to include Content only for now and manually add rest of the plugins as I require to complete the project.

Then it will ask you to select the Linting Tool, It's always ESLint for me. After that, the setup will ask you to set up a testing framework. Since this is not a huge project, I wouldn't recommend that. So I'll select None.

Is your page is a Single Page App (SPA) or will it contain routing? We are building a blog so obviously there will be routing. So, go ahead and select Universal

Next question is very crucial, In this post, we will be building a completely static website. Therefore you will select Static in this stage.

Next is Development tools, don't worry about this, and select jsconfig.json, Then I will ask if you have a git version control system (In my case, I have a git where you can find all the code to this project) But you can select None if you want.

Once this step is completed, change directory into that folder. In my case, it's MyBlog

cd MyBlog

At this stage, you have created a nuxt.js application. You can see this by writing the following code in your terminal.

npm run dev

Usually, it will run your website on [localhost port 3000](http://localhost:3000 "Localhost 3000"). You can see that in your terminal.


This is just the beginning, we will create a functioning blog from here.

Folder Structure

When you open the newly created folder in your text editor, you'll see something like this. Most of these folders are not that important for this project. Let me explain the ones that you'll use.

1. Assets: This folder will

Markdown Frontmatter

Fetching Articles

Code Formatting

Latex Support

Date Formatting & Read Time

Add Image Support

Youtube Embed

Generate RSS

Generate Sitemap

A perceptron can be represented as a function that takes some input and gives off some output. This function any perceptron performs is same in all scenario, no matter what that perceptron is intended to do.

A perceptron takes the inputs, which is an (n+1)-dimensional vector containing n input parameters and 1 bias variable and multiplies with the weight vector (an array of size n+1 and the first element being 1) and wraps it around an activation function and that becomes the output of the perception.

If that goes straight through your mind, let me explain.

No matter what’s the learning problem is, a machine learning algorithm always plots the inputs in a graph and try to find the optimum line that passes through this plotted points.

A perceptron matches this behavior by multiplying inputs and weights and activating the product. Mathematically, this will only work if the first element of the weight vector is 1 and the first element of the input vector is the bias parameter. Consider the following code, that demonstrates what you just read in this paragraph.

# W is for weights vector.
# X is for input vector.
X = [2,3,1,3,5,7,8,5] # Just some input vector.
# The first element ,ie, X[0] is the bias parameter

W = np.random.rand(len(X))
w[0] = 1
# This will initialize the W vector with random numbers
# and sets the first element ,ie, W[0] to 1

Now before we move further, let us talk about some machine learning jargon. We’ll start with

Hypothesis Function

A “Hypothesis Function” is nothing but the mathematical function of the line or curve that fits the given data-set. A function that “fits” the data-set is the one that connects most of the points in the data-set with minimum error. The mathematical function can be demonstrated using the following code.

# np is an object of NumPy class
def hypothesis(X,W):
    return np.matmul(X,np.transpose(W))

The above hypothesis function gives any real number which is great when you are doing some regression task but when you are performing a classification task, you have to enclose the result of X*W in some function that has a range from 0 to 1. One function that is common and used a lot for this task is called “Sigmoid function”, mathematical code for that is as follows:

# Y is the actual values
# corresponding to X in the data-set
def cost(X,Y):
    sum = 0
    for i,x in enumerate(hypothesis(X)):
        sum += (x - Y[i])**2
    return sum/(2*len(X))

The above code demonstrate the cost function for a regression task but if you want to find the error in a classification task, you have to consider the following code.

def cost(X,Y):
    sum = 0
    for i,x in enumerate(hypothesis(X)):
        sum += -(Y[i]*np.log(sigmoid(x))+(1-Y[i])*np.log(1-sigmoig(x)))
    return sum/len(X)

If you want to learn in depth about cost functions, you can refer this video by Andrew Ng.

Perceptron algorithm

The below code demonstrates an algorithm to find the optimum weights for our data set that can plot a line which best divides or fits the training data set, X.

def getOptimumWeights(X,Y,learn_rate,epoch):

    W = np.rand.random(len(X))
    W[0] = 1
    i = 0
    while i > epoch:
        i += i
        error = cost(X,Y)
        for w in W[1:]:
            w = w - learn_rate*error
    return W

We first create a variable W and assign it to a list of random number with length equal to your number of training set. Then assign first element of W to 1. This is because W[0] corresponds to the Bias factor, that is, if you plot the hypothesis function as a graph then Bias factor decides the y point at x = 0.

Then we iteratively update the list W according to the learning rate and and error.

Learning Rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Take a look at this video.

Now that is done, let’s concentrate on the actual perceptron algorithm, a simple rule of a perceptron or any neural network is take the input, multiply by weights, add the bias and activate. As stated above, we can simulate the add the bias step be fixing the first element of W as 1 and the first element of training data set, X as the bias. The need of activate step or activation is to add a sense of non-linearity to the line that we want to fit, this will improve the accuracy of our model to a great extend and our perceptron or neural network can learn a wide variety of things. Consider the following code.

# X*W is basically equal to X*W + b,
# Remember take the input, multiply by weights, add the bias and activate
def perceptron(X,W):
    # We are using sigmoid function to activate our hypothesis
    output = sigmoid(X*W)
    return output

before using this this function we need to do some pre-processing that is to train the weights vector W, remember the function we made called getOptimumWeights that function does this task. Theoretically, you can use any real number as the learning rate, but ideally, you have to use something that is neither too high nor too low. I always use 0.01 or 0.005 as the learning rate.

Using Perceptron as Logical Gates

The best thing about neural networks or perceptron is that you can teach a computer to do any task by giving it some inputs and corresponding outputs.

First, lets teach this perceptron to learn and tell the outputs of an AND gate.

X = [[0,0],[0,1],[1,0],[1,1]]
Y = [0,0,0,1]

W = getOptimumWeights(X,Y,0.005)
output = perceptron(X,W)

for o in output
    if o < 0.5:
        print(0)
    else:
        print(1)

We created X and Y variables as per above image and passed it through previously created optimisation function. This gave us the Weights to predict the outputs using perceptron.

The output for above pipeline will be as follows.

0
0
0
1

This way, you can make any logical gates and when you combine multiple perceptrons, you’ll end up with a full fledged Neural Network that can be used to learn large data sets such as stock market, or text emotions if you want to make a chat bot that looks just like human chats. You can make Siri or Cortana using Neural Networks. Once you start using NNs or Machine Learning the possibilities are endless.

That is all for this post. If you like my content, make sure to subscribe to this blog and also share this post to your tech community.

Thank You.

Machine learning (ML) is a category of algorithm that allows software applications to become more accurate in predicting outcomes without being explicitly programmed. The basic premise of machine learning is to build algorithms that can receive input data and use statistical analysis to predict an output while updating outputs as new data becomes available.

Basically, machine learning is the way that makes your smartphones “smart”, You often see in big corporate conventions they use Machine Learning and AI interchangeably even though these two topics are very different but connect in many ways. The goal of AI or Artificial Intelligence is to make a machine or robot that can mimic the capabilities of a human mind, which of course includes learning abilities but also understanding texts and languages (Natural Language Processing), ability to comprehend images and videos (Computer Vision) and other things a human can do.

[Tom M. Mitchell](https://en.wikipedia.org/wiki/Tom_M._Mitchell "Tom M. Mitchell") provided a widely quoted, more formal definition of the algorithms studied in the machine learning field: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”

In other words, Machine Learning is a part of AI that deals only with the learning abilities of a machine. All the algorithms and techniques that are dedicated to the task of teaching a machine to learn based on past experiences like Linear Regression, Support Vector Machines, Logistic Regression, Gradient Descent, etc all come under the field of Machine Learning.

But when these companies like Google or Apple wants to sell something, they use the term Artificial Intelligence or AI and when they talk about their projects to engineers and other experts, they use the term Machine Learning and Neural Networks.

Machine Learning can be split into three categories: Supervised Learning, Unsupervised Learning and Reinforcement Learning.

Supervised Learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from labelled training data consisting of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and the desired output value (also called the supervisory signal). A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. An optimal scenario will allow for the algorithm to correctly determine the class labels for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a “reasonable” way.

Unsupervised learning is a branch of machine learning that learns from test data that has not been labelled, classified or categorized. Instead of responding to feedback, unsupervised learning identifies commonalities in the data and reacts based on the presence or absence of such commonalities in each new piece of data.

Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. The problem, due to its generality, is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, statistics and genetic algorithms. In the operations research and control literature, reinforcement learning is called approximate dynamic programming, or neuro-dynamic programming. The problems of interest in reinforcement learning have also been studied in the theory of optimal control, which is concerned mostly with the existence and characterization of optimal solutions, and algorithms for their exact computation, and less with learning or approximation, particularly in the absence of a mathematical model of the environment. In economics and game theory, reinforcement learning may be used to explain how equilibrium may arise under bounded rationality. In machine learning, the environment is typically formulated as a Markov Decision Process (MDP), as many reinforcement learning algorithms for this context utilize dynamic programming techniques. The main difference between the classical dynamic programming methods and reinforcement learning algorithms is that the latter does not assume knowledge of an exact mathematical model of the MDP and they target large MDPs where exact methods become infeasible.

We’ll be looking into algorithms for each of these categories in separate posts (due to SEO reasons)

If you paid attention to this post, I wrote something about Neural Network

What is a Neural Network?

Artificial neural networks (ANN) or Neural Network (NN) are computing systems vaguely inspired by the biological neural networks that constitute animal brains. The neural network itself is not an algorithm, but rather a framework for many different machine learning algorithms to work together and process complex data inputs. Such systems “learn” to perform tasks by considering examples, generally without being programmed with any task-specific rules. For example, an image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labelled as “cat” or “no cat” and using the results to identify cats in other images. They do this without any prior knowledge about cats, for example, that they have fur, tails, whiskers and cat-like faces. Instead, they automatically generate identifying characteristics from the learning material that they process.

Naturally, a question will arise in all Machine Learning beginners that if we have ML algorithms that can do almost all work with great accuracy, what is the need of Neural Networks?

One of my favorite Youtuber 3Blue1Brown has made a video about this

You can watch this video if you have 20 mins to spare, but I’ll try to summarize this video as accurate and as simple as possible.

Basically, a Neural Network, unlike a traditional Machine Learning algorithm, resembles more to the actual brain and the cells in a brain. And like the brain, a neural network is capable of learning different tasks. But if we use Machine Learning algorithms then we have to develop different algorithms for different tasks.

A neuron in a neural network is known as a Perceptron, a Perceptron just like any function takes some input and gives some output. And in machine learning, a perceptron is used in supervised learning both regression and classification.

In statistical modelling, regression analysis is a set of statistical processes for estimating the relationships among variables. It includes many techniques for modelling and analyzing several variables when the focus is on the relationship between a dependent variable and one or more independent variables.

classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.

We will learn about Perceptrons in the next post, in that post, we will also discuss a way with which we can make basic logic gates (AND, OR) and an optimization algorithm called Gradient Descent.

If you like my blog, then make sure that you follow this blog and also leave a like and share this post with your friends and colleagues. If you have any query then please leave a comment, I’ll reply to them as soon as possible.

Thank You.

In this installment of our series TIL (Today I Learned), I will be writing about the story of how I was able to build a decent Quiz bot on telegram in less than 1 hour. (I wrote this post and watched some Youtube videos in the remaining 2 hours of my day)

As any software developer in this planet, I first went to GitHub to see if there is any code that I can straight up copy. But unfortunately, then once that were decent was like 2 years old. I started reading them and got to know about this node.js library for Telegram Bots API called Telegraf.

Then I quickly built a fresh folder in my Projects directory and initiated npm.

npm init -y

After that, I installed telegraf (obviously)

npm i telegraf

Copy pasted the example code provided by Telegraf and realised there is a problem.

const { Telegraf } = require('telegraf')

const bot = new Telegraf(process.env.BOT_TOKEN)
bot.start((ctx) => ctx.reply('Welcome!'))
bot.help((ctx) => ctx.reply('Send me a sticker'))
bot.on('sticker', (ctx) => ctx.reply('👍'))
bot.hears('hi', (ctx) => ctx.reply('Hey there'))
bot.launch()

I don't actually have a "BOT TOKEN", the search of which led to me to this Telegram Bot called BotFather which made it incredibly easy to get the BOT API Token.

After that I created an environment variable called BOT_TOKEN and assigned the Telegram token to it. Then I started the node serer by

node index.js

Searched my BOT on Telegram and ...

So, 10 mins into code I have a basic chat bot ready. Since this was just a proof of concept, I didn't want to spend time of setting up database and stuff so I created a dictionary of current players and questions.

const questions = [
    {
        "question": "The tree sends down roots from its branches to the soil is know as:",
        "o1": "Oak",
        "o2": "Pine",
        "o3": "Banyan",
        "answer": "o3"
    }
]

let players = {}

For I had 5 questions, but for this blog, there is only one. Also, I will initiate the players dynamically later on.

In telegram, you can send predefined commands such as /start or /help and the chatbot manager can send appropriate replies. For this bot I decided, I will send this message whenever a new user sends me /start command.

bot.start((ctx)=>{
    ctx.reply(`Hello, Are you ready to start the quiz?\nThere will be ${questions.length} question, each scoring 1 points.`)

    const logo = fs.readFileSync('img/logo.jpg')

    ctx.telegram.sendPhoto(ctx.chat.id, {
        source: logo
    }).catch(reason => {
        console.log(reason)
    })
})

Basically, I wrote a custom function on the example code that telegraf provided. Now the chatbot will behave like this.

The code you're gonna see next is shit and redundant, that's what being a senior develop is all about.

~ Chris (10 years of experience in Software Engineering)

Whenever the bot gets a message from a user, I want the manager to check if this player is new or have registered to the system. This can be done using a simple if condition since we are using in memory dictionaries.

bot.on('message', (ctx) => {
    let userId = ctx.message.from.id;
    if (userId in players){
        if (players[userId]['last_question'] >= questions.length){
            quiz_end_message = `You've completed the trivia ❤️❤️❤️, you scored ${players[userId].score} out of ${questions.length}.`

            const keyboard = Markup.inlineKeyboard([
                Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com/'),
              ])

            ctx.reply(quiz_end_message, Extra.markup(keyboard))
        } else {
            const keyboard = Markup.inlineKeyboard([
                [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
                [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
                [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
              ])

            ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
        }
    } else {
        players[userId] = {
            "last_question": 0,
            "last_answer_time": new Date().getTime(),
            "score": 0
        }

        console.log(userId)

        const keyboard = Markup.inlineKeyboard([
           [Markup.callbackButton(questions[0].o1,"o1")],
            [Markup.callbackButton(questions[0].o2,"o2")],
            [Markup.callbackButton(questions[0].o3,"o3")]
          ])
          ctx.reply(questions[0].question, Extra.markup(keyboard))
    }
})

Just read the code, I'll explain what's going on in a minute.

First of all, we can get the ID of the user who just messaged us using

ctx.message.from.id

then we checked if this id is already present in our memory, if it doesn't that means the else block will get executed, where we initiate the player and send the first question to the user. The keyboard variable defines the option buttons for the quiz, which you can see in the image below.

There are many kinds of Buttons that you can see on Telegraf docs, but I only needed callbackButton because using this I can get to know which button the user clicked.

But if that condition were true, that is, if a user has already registered. This I check if the player has completed the game in which case I just replied the score with a link.

In telegraf, you can define custom actions using callbackButton the one you saw above.

Markup.callbackButton(questions[0].o1,"o1")

The second parameter is the one that defines the name of the action.

bot.action('o1', async(ctx) => {
    let userId = ctx.callbackQuery.message.chat.id;

    updateScore(userId, 'o1')

    if (players[userId].last_question >= questions.length){
        quiz_end_message = `You've completed the trivia ❤️❤️❤️, you scored ${players[userId].score} out of ${questions.length}.`
        const keyboard = Markup.inlineKeyboard([
            Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com'),
          ])

        ctx.reply(quiz_end_message, Extra.markup(keyboard))
    } else {
        const keyboard = Markup.inlineKeyboard([
            [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
          ])

        ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
    }

    ctx.deleteMessage()
})

This is how we create a custom action, this callback function is called when a user clicks option 1 or the first button. I didn't wanted to handle multiple questions on a screen which led to a possibility that a user might click on a option for previous question. So, I just deleted the question once it's score is checked and updated the score.

function updateScore(playerId ,option){
    const last_question =  players[playerId].last_question;

    console.log(players)

    players[playerId]['last_question']++;

    if (questions[last_question].answer === option){
        players[playerId].score++;
    }
}

With each click of the options, I am also checking if the user have completed the quiz, in which case I sent them the score and a link.

So that is it, it took me about 50 mins to build this and here is the entire code.

const { Telegraf } = require("telegraf")
const Extra = require('telegraf/extra')
const Markup = require('telegraf/markup')

const fs = require('fs')

const bot = new Telegraf(process.env.BOT_TOKEN)
bot.start((ctx)=>{
    ctx.reply(`Hello, Are you ready to start the quiz?\nThere will be ${questions.length} question, each scoring 1 points.`)

    const logo = fs.readFileSync('img/logo.jpg')

    ctx.telegram.sendPhoto(ctx.chat.id, {
        source: logo
    }).catch(reason => {
        console.log(reason)
    })
})

let players = {}

let questions = [
    {
        "question": "The tree sends down roots from its branches to the soil is know as:",
        "o1": "Oak",
        "o2": "Pine",
        "o3": "Banyan",
        "answer": "o3"
    },
    {
        "question": "Electric bulb filament is made of ",
        "o1": "Copper",
        "o2": "Lead",
        "o3": "Tungsten",
        "answer": "o3"
    },
    {
        "question": "Which of the following is used in pencils?",
        "o1": "Graphite",
        "o2": "Lead",
        "o3": "Silicon",
        "answer": "o1"
    }, 
    {
        "question": "The speaker of the Lok Sabha can ask a member of the house to stop speaking and let another member speak. This phenomenon is known as :",
        "o1": "Crossing the floor",
        "o2": "Yielding the floor",
        "o3": "Calling Attention Motion",
        "answer": "o2"
    },
    {
        "question": "The Comptroller and Auditor General of India can be removed from office in like manner and on like grounds as :",
        "o1": "High Court Judge",
        "o2": "Prime Minister",
        "o3": "Supreme Court  Judge",
        "answer": "o3"
    },
]

bot.on('message', (ctx) => {
    let userId = ctx.message.from.id;
    if (userId in players){
        if (players[userId]['last_question'] >= questions.length){
            quiz_end_message = `You've completed the trivia      🥳     🥳, you scored ${players[userId].score} out of ${questions.length}.`

            const keyboard = Markup.inlineKeyboard([
                Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com/'),
              ])

            ctx.reply(quiz_end_message, Extra.markup(keyboard))
        } else {
            const keyboard = Markup.inlineKeyboard([
                [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
                [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
                [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
              ])

            ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
        }
    } else {
        players[userId] = {
            "last_question": 0,
            "last_answer_time": new Date().getTime(),
            "score": 0
        }

        console.log(userId)

        const keyboard = Markup.inlineKeyboard([
           [Markup.callbackButton(questions[0].o1,"o1")],
            [Markup.callbackButton(questions[0].o2,"o2")],
            [Markup.callbackButton(questions[0].o3,"o3")]
          ])
          ctx.reply(questions[0].question, Extra.markup(keyboard))
    }
})

function updateScore(playerId ,option){
    const last_question =  players[playerId].last_question;

    console.log(players)

    players[playerId]['last_question']++;

    if (questions[last_question].answer === option){
        players[playerId].score++;
    }
}

bot.action('o1', async(ctx) => {
    let userId = ctx.callbackQuery.message.chat.id;

    updateScore(userId, 'o1')

    if (players[userId].last_question >= questions.length){
        quiz_end_message = `You've completed the trivia      🥳     🥳, you scored ${players[userId].score} out of ${questions.length}.`
        const keyboard = Markup.inlineKeyboard([
            Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com/'),
          ])

        ctx.reply(quiz_end_message, Extra.markup(keyboard))
    } else {
        const keyboard = Markup.inlineKeyboard([
            [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
          ])

        ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
    }

    ctx.deleteMessage()
})

bot.action('o2', async(ctx) => {
    let userId = ctx.callbackQuery.message.chat.id;

    updateScore(userId, 'o2')

    if (players[userId].last_question >= questions.length){
        quiz_end_message = `You completed the trivia, you scored ${players[userId].score} out of ${questions.length}.`
        const keyboard = Markup.inlineKeyboard([
            Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com/'),
          ])

        ctx.reply(quiz_end_message, Extra.markup(keyboard))
    } else {
        const keyboard = Markup.inlineKeyboard([
            [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
          ])

        ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
    }

    ctx.deleteMessage()
})

bot.action('o3', async(ctx) => {
    let userId = ctx.callbackQuery.message.chat.id;

    updateScore(userId, 'o3')

    if (players[userId].last_question >= questions.length){
        quiz_end_message = `You completed the trivia, you scored ${players[userId].score} out of ${questions.length}.`
        const keyboard = Markup.inlineKeyboard([
            Markup.urlButton('Click here to get the Prizes', 'https://idiomaticprogrammers.com/'),
          ])

        ctx.reply(quiz_end_message, Extra.markup(keyboard))
    } else {
        const keyboard = Markup.inlineKeyboard([
            [Markup.callbackButton(questions[players[userId]['last_question']].o1,"o1")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o2,"o2")],
            [Markup.callbackButton(questions[players[userId]['last_question']].o3,"o3")]
          ])

        ctx.reply(questions[players[userId]['last_question']].question, Extra.markup(keyboard))
    }

    ctx.deleteMessage()
})

bot.launch()

That is what I learned today.