The internet isn't a single, monolithic cloud. It’s a massive, interconnected patchwork of thousands of independent networks, and the most fundamental building block of this global structure is the Autonomous System (AS).

Think of an AS as a digital country 🗺️. It has its own borders, makes its own rules for how traffic moves internally (its routing policy), and is managed by a single entity—be it an ISP like WorldLink, a tech giant like Google, or a university.

For one of these digital countries to participate in global traffic exchange, it needs two critical identifiers:

• ASN (Autonomous System Number): This is the country's official, unique code on the world map (e.g., AS17501 for WorldLink). It’s the nameplate used by the internet's global GPS—the Border Gateway Protocol (BGP)—to identify the network.

• IP Prefixes: These are the street addresses within that country (e.g., 103.23.140.0/22). They represent the blocks of "digital real estate" that the AS owns and manages.

Without an ASN, a network has no identity on the world stage. Without IP prefixes, it has no territory to announce. An AS needs both to be a recognized and functioning part of the global internet.

Who Assigns ASNs and IPs?

The distribution of these resources follows a strict hierarchy:

1. IANA (Internet Assigned Numbers Authority): Maintains the global pool of IP space and ASN ranges.

2. RIRs (Regional Internet Registries): Allocate resources to organizations within their regions. Nepal and the Asia-Pacific region fall under APNIC.

3. ISPs and LIRs (Local Internet Registries): Request and manage ASNs and IP addresses from their RIR.

This structure ensures uniqueness and prevents conflicts across the global internet.

Real-World ASN Examples and Their IP Prefixes

Internet Hierarchy

The internet has an informal structure based on a network's scale and how it connects to the world.

Traditional ISP Tiers

• Tier 1: The global backbones of the internet. Their networks are so extensive they don't pay for access, instead peering freely with all other Tier 1s. They're like major international airlines with routes to every continent.

• Tier 2: Large national or regional providers. They peer extensively to save costs but must still buy IP Transit from a Tier 1 to ensure their customers have full global reach. They are the national airlines that partner with global ones for overseas flights.

• Tier 3: Smaller, "last-mile" providers that deliver internet directly to homes and businesses. They almost exclusively purchase transit from larger ISPs. They're the local taxi service that takes you to the main airport.

Hyperscalers

Hyperscalers are large technology companies operating massive private networks, often larger than traditional Tier 1 ISPs. Unlike the tiered ISP model, their networks are built to support global cloud platforms, large-scale content delivery, and online services.

Key characteristics:

• Operate their own global backbones interconnecting data centers worldwide.

• Peer directly with thousands of ISPs to deliver services efficiently.

• Minimize dependence on transit providers.

Examples of hyperscalers include:

• Google (AS15169): Operates Google Cloud, YouTube, and Gmail.

• Amazon (AS16509): Runs Amazon Web Services (AWS).

• Microsoft (AS8075): Runs Azure and Microsoft 365.

• Meta (AS32934): Operates Facebook, Instagram, and WhatsApp.

• Netflix (AS2906): Operates a global content delivery network for video streaming.

• Cloudflare (AS13335): Provides CDN, security, and edge computing services.

These companies invest heavily in infrastructure and connect directly to ISPs and IXPs worldwide to improve performance and reduce costs. You can go to https://ipinfo.io/. And check the IP blocks they control and other details.

Connections Between Networks

Every Autonomous System (AS) must connect with others to exchange traffic, since no single AS can reach the entire internet on its own. These connections typically fall into two main business relationships:

• Transit:
  Transit is a paid service where one AS (the provider) agrees to carry all of another AS’s traffic to the rest of the internet. In practice, this gives the customer network full global reachability. Smaller ISPs, enterprises, or payment platforms often buy transit from larger providers. Pricing is usually based on bandwidth capacity (e.g., per Mbps or Gbps). Transit agreements create a hierarchical internet structure, with smaller ASes “upstream” of larger ones.

• Peering:
  Peering is a mutual agreement where two ASes exchange traffic only between their respective customers, without payment. The main motivation is efficiency—peering reduces latency and avoids transit costs. However, peering is not guaranteed; networks peer only when both sides see mutual benefit, such as high traffic volume between them or geographic proximity.

Internet Exchange Points (IXPs)

Most peering relationships are established at Internet Exchange Points (IXPs). An IXP is a physical infrastructure (typically a high-capacity Ethernet switch) where multiple networks interconnect through a shared fabric. Instead of setting up dozens or hundreds of private links, each participant only needs one connection to the IXP, and from there they can peer with many others.

Benefits of IXPs include:

• Reduced Cost: One port at the IXP replaces many separate links, lowering operational and bandwidth expenses.

• Lower Latency: Traffic between local or regional networks can be exchanged directly, avoiding long detours through upstream providers.

• Scalability: Networks can easily add or remove peers without reconfiguring complex topologies.

• Ecosystem Growth: IXPs often act as hubs, attracting CDNs (like Cloudflare, Akamai), hyperscalers (Google, AWS, Microsoft), financial platforms, and local ISPs, which further strengthens local internet resilience.

Well-known IXPs include LINX (London Internet Exchange), DE-CIX (Germany), and AMS-IX (Amsterdam). In Nepal, NPIX (Nepal Internet Exchange) plays this role by connecting local ISPs and service providers, keeping domestic traffic within the country.

ISP Startup Checklist ✅

Building an ISP involves both regulatory approval and technical infrastructure. The core steps are:

1. Legal Setup

   • Register a company.

   • Obtain an ISP license from the national regulator (in Nepal: NTA).

2. Identity (ASN and IPs)

   • Apply to your RIR-APNIC(Asia Pacific Network Information Centre) for an ASN and IP address blocks.

   • Expect primarily IPv6 allocations, as IPv4 is scarce.

3. Point of Presence (PoP)

   • Lease rack space in a carrier-neutral data center.

   • Deploy:

     • Carrier-grade routers capable of BGP and handling large routing tables.

     • Switches for internal aggregation.

     • Servers for DNS, RADIUS (authentication), and monitoring.

   • Ensure redundancy in power, cooling, and connectivity.

4. Transit Connectivity

   • Sign a contract with an upstream provider.

   • Establish cross-connects to their equipment in the same facility.

   • Configure external BGP sessions for global reachability.

5. Peering

   • Join the regional IXP (in Nepal: NPIX).

   • Establish peering sessions with local ISPs, content networks, and hyperscalers present at the exchange.

   • Reduce transit costs and improve latency for local traffic.

6. Operations

   • Deploy monitoring and logging systems.

   • Establish network security policies.

   • Set up customer support and provisioning systems.

Next Steps

Now that we’ve covered the foundation of ASNs, IP prefixes, transit, peering, and IXPs, the next critical topic is BGP (Border Gateway Protocol).

BGP acts as the duct tape of the internet, holding together the complex web of networks and enabling global connectivity. It allows networks to announce their prefixes, learn routes from other ASes, and make intelligent decisions about the best paths for traffic.

In the next blog, we’ll explore:

• How BGP sessions are established.

• The difference between internal (iBGP) and external (eBGP) sessions.

• Visualization of BGP works.

• Real-world examples of major outages caused by BGP hijacking or misconfigurations, demonstrating why careful BGP management is critical for network stability.

This will give a practical understanding of how the internet’s routing backbone works and why even small mistakes can have wide-reaching effects.

Every device on the internet is part of a smaller network first — your phone connected to Wi-Fi, your laptop plugged into Ethernet, or even a server inside a data center rack. These local networks are the foundations of the global internet. Without them, nothing moves.

At the heart of these small networks is the same problem: how does one device know how to send data to another? That’s where networking begins.

ARP – The First Step in Talking to Your Router

So everything starts with a basic ARP request, a process that tools like arp-scan are built to exploit. The primary purpose of ARP (Address Resolution Protocol) is to map an IP address to its corresponding MAC address so that communication can actually happen at the data link (Ethernet) layer.

Step 1: Subnet Masking on the Source Machine

• When your computer wants to send a packet, it first compares the destination IP with its own IP and subnet mask.

  • How IP masking works:

    1. Your computer performs a bitwise AND between its own IP and the subnet mask to get the network address.

    2. It also performs a bitwise AND between the target IP and the same subnet mask.

    3. If the resulting network addresses are equal, the IP is in the same subnet. Otherwise, it’s outside your local network.

• This determines if the destination is on the same local network (on-link) or off-network (needs a gateway).

Step 2a: Destination is On-Link

• If the destination is in the same subnet:

  1. The OS knows it can reach the host directly.

  2. It checks the ARP cache for the MAC of the destination IP.

  3. If the MAC isn’t in the cache, it sends an ARP request:
     "Who has IP X? Tell me your MAC!"

  4. The destination machine replies with its MAC.

  5. Packet is wrapped in an Ethernet frame with:

     • Destination MAC = target machine’s MAC

     • Destination IP = target machine’s IP

• Packet is sent directly to the destination.

# ARP scan to check available hosts in local network.
[nix-shell:~]$ sudo arp-scan --localnet
Interface: wlp0s20f3, type: EN10MB, MAC: 94:b6:09:66:b8:27, IPv4: 192.168.1.111
Starting arp-scan 1.10.0 with 256 hosts (https://github.com/royhills/arp-scan)
192.168.1.137    40:e1:e4:0e:0a:81    Nokia Solutions and Networks GmbH & Co. KG
192.168.1.81    96:94:6c:ea:3b:3f    (Unknown: locally administered)
192.168.1.130    70:08:94:3b:9e:51    (Unknown)
192.168.1.124    a0:b3:39:62:0c:cd    (Unknown)
192.168.1.254    c8:9c:bb:52:bc:80    (Unknown)

5 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.10.0: 256 hosts scanned in 1.970 seconds (129.95 hosts/sec). 5 responded

# Check entries in arp cache
[nix-shell:~]$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.19.0.2               ether   02:42:ac:13:00:02   C                     br-bd6d6f551404
172.18.0.2               ether   02:42:ac:12:00:02   C                     br-5d681ffed7e7
192.168.1.254            ether   c8:9c:bb:52:bc:80   C                     wlp0s20f3

# Ping one of the device in the localnetwork
[nix-shell:~]$ ping 192.168.1.137
PING 192.168.1.137 (192.168.1.137) 56(84) bytes of data.
64 bytes from 192.168.1.137: icmp_seq=1 ttl=64 time=4.92 ms
64 bytes from 192.168.1.137: icmp_seq=2 ttl=64 time=3.09 ms
^C
--- 192.168.1.137 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 3.087/4.002/4.918/0.915 ms

# Check the ARP cache. You can find the newly pinged device.
[nix-shell:~]$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.19.0.2               ether   02:42:ac:13:00:02   C                     br-bd6d6f551404
172.18.0.2               ether   02:42:ac:12:00:02   C                     br-5d681ffed7e7
192.168.1.137            ether   40:e1:e4:0e:0a:81   C                     wlp0s20f3
192.168.1.254            ether   c8:9c:bb:52:bc:80   C                     wlp0s20f3

Step 2b: Destination is Off-Link

• If the destination is outside the local subnet:

  1. The OS consults the routing table (ip route show).

  2. If there’s a specific route for the network, the packet is sent to the next hop specified there.

  3. Otherwise, it uses the default route (usually your router/gateway).

  4. The OS performs ARP for the gateway IP, finds the router’s MAC, and wraps the packet in an Ethernet frame with:

     • Destination MAC = router MAC

     • Destination IP = actual target IP

  5. Packet is sent to the router.

# Get the IP address
[nix-shell:~]$ dig +short google.com
172.217.26.46

# Check the Route table
[nix-shell:~]$ ip route show
default via 192.168.1.254 dev wlp0s20f3 proto dhcp src 192.168.1.111 metric 600 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-5d681ffed7e7 proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev br-bd6d6f551404 proto kernel scope link src 172.19.0.1 
172.20.0.0/16 dev br-b46a2861832b proto kernel scope link src 172.20.0.1 linkdown 
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.111 metric 600 
192.168.49.0/24 dev br-5ca7647dd12d proto kernel scope link src 192.168.49.1 linkdown 

# No specific route for google's IP. So arp the default route
[nix-shell:~]$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.19.0.2               ether   02:42:ac:13:00:02   C                     br-bd6d6f551404
172.18.0.2               ether   02:42:ac:12:00:02   C                     br-5d681ffed7e7
192.168.1.137            ether   40:e1:e4:0e:0a:81   C                     wlp0s20f3
192.168.1.254            ether   c8:9c:bb:52:bc:80   C                     wlp0s20f3

# here we can see the MAC address of the router to be c8:9c:bb:52:bc:80

Step 3: Routers Along the Path

• Each router repeats the same process:

  1. Checks its routing table for the next hop.

  2. If the next hop is on a connected network, ARPs for the MAC.

  3. If not, forwards to another router toward the destination.

• At each hop, the MAC addresses change, but the IP addresses remain unchanged.

[nix-shell:~]$ dig +short google.com 
172.217.26.46

[nix-shell:~]$ traceroute 172.217.26.46
traceroute to 172.217.26.46 (172.217.26.46), 30 hops max, 60 byte packets
 1  * _gateway (192.168.1.254)  3.753 ms  3.743 ms
 2  27.34.24.1 (27.34.24.1)  6.681 ms  9.079 ms  8.718 ms
 3  be-82-8.45.gwc-ndc-core-01.wlink.com.np (202.79.45.8)  10.175 ms  10.165 ms  10.156 ms
 4  ae-20-136.41.gwj-htda-core-01.wlink.com.np (202.79.41.136)  10.077 ms  10.860 ms  10.851 ms
 5  ae-21-139.41.gwj-btwl-core-01.wlink.com.np (202.79.41.139)  12.951 ms  12.941 ms  12.932 ms
 6  ae52-ipt-bhwa-01.wlink.com.np (72.9.128.67)  12.908 ms  11.520 ms  11.491 ms
 7  * * *
 8  142.250.174.2 (142.250.174.2)  30.058 ms  24.876 ms  31.027 ms
 9  192.178.83.35 (192.178.83.35)  38.327 ms 192.178.83.245 (192.178.83.245)  37.036 ms 192.178.83.35 (192.178.83.35)  31.181 ms
10  142.251.49.121 (142.251.49.121)  30.083 ms 142.251.49.115 (142.251.49.115)  30.032 ms 142.251.49.121 (142.251.49.121)  30.047 ms
11  nrt12s17-in-f46.1e100.net (172.217.26.46)  23.404 ms  26.114 ms  20.930 ms

• Hop 1 – Your home router (default gateway). The packet leaves your local network.

• Hops 2–6 – Routers inside ISP’s network (wlink.com.np) forwarding the packet toward the global internet.

• Hop 7 – No response (* * *), a router that blocks traceroute probes. But packets passes through.

• Hops 8–10 – Global internet routers, moving the packet toward Google. Multiple IPs indicate load balancing.

• Hop 11 – Final destination: the Google server (172.217.26.46), where the packet arrives successfully.

Step 4: Destination Network

• When the packet finally reaches the router directly connected to the destination network:

  1. The router performs subnet masking to see the destination IP is on its local subnet.

  2. It performs ARP to find the MAC of the actual destination machine.

  3. Wraps the packet with that MAC and sends it to the server.

• At this point, the packet has reached the actual host, and the process is complete.

Next Steps: Building the Networks

Our traceroute gave us a direct glimpse inside the infrastructure of an ISP, wlink.com.np, and the global backbone of Google. This raises a crucial question: What elevates a network from a simple collection of routers to an official Internet Service Provider? How does an entity like WorldLink get the authority to manage its own segment of the internet, with its own unique identifiers and blocks of public IP addresses?

In the next part of this series, we will answer that question. We’ll move from analyzing packets to building the networks that carry them, providing a practical, step-by-step checklist on how to build your own ISP from the ground up.

If you want a proper tutorial by a professional, you can check out Hussein Nassar’s Network Routing video. This video was one of the inspiration for creating this blog series.

This blog is generated as a usecase of this other blog(Automating blog creation with own agent vs mcp) that helped me automate the writing of this blog.

Alright, folks. Let's talk about Python concurrency. And by "talk," I mean "I'm going to rant about it while showing you some actual data. We're diving deep into the GIL, threads (with and without that pesky GIL), processes, and async, and we're going to benchmark the living hell out of them. Prepare for graphs. Prepare for truth. Prepare for me to question my life choices.

The GIL: Python's Arch-Nemesis (or, Why Your CPU Isn't Doing What You Think It's Doing)

The Global Interpreter Lock (GIL). You've heard the whispers. The boogeyman of Python performance. In short, it's a mutex that allows only one thread to hold control of the Python interpreter at any given time. This means that even if you have a shiny new multi-core CPU, your Python code might be running on only one core at a time.

Why does it exist? Historical reasons. It simplifies memory management and makes C extensions easier to write. Is it ideal? Absolutely not. But it's what we've got (until now... more on that later).

The Contenders: Threads, Processes, and Async (Oh My!)

We're going to pit these concurrency methods against each other in two scenarios: CPU-bound tasks and I/O-bound tasks.

• Threads (GIL Enabled): The standard Python threading library. Good for I/O, terrible for CPU.

• Threads (GIL Disabled - Experimental): Python 3.13 (experimental) allows you to disable the GIL on a per-interpreter basis. Let's see if it lives up to the hype!

• Processes (Multiprocessing): Spawns separate Python processes, each with its own interpreter and memory space. Bypasses the GIL, great for CPU-bound tasks, but has overhead.

• Asyncio: Single-threaded, concurrent execution using coroutines. Excellent for I/O-bound tasks, requires careful coding to avoid blocking the event loop.

Benchmarking Setup: The Nitty-Gritty (aka, Show Me the Code!)

import argparse
import asyncio
import sys

from runners import (
    run_threading_cpu,
    run_multiprocessing_cpu,
    run_threading_io,
    run_asyncio_io,
)

def check_gil_enabled():
    import sys
    try:
        return sys._is_gil_enabled()
    except AttributeError:
        return True

def main():
    parser = argparse.ArgumentParser(description="Benchmark CPU/IO tasks with concurrency.")
    parser.add_argument("--task", choices=["cpu", "io"], required=True, help="Type of benchmark to run.")
    parser.add_argument("--method", choices=["threading", "multiprocessing", "asyncio"], required=True, help="Concurrency method.")
    parser.add_argument("--workers", type=int, default=4, help="Number of workers/threads/processes/tasks.")
    parser.add_argument("--n", type=int, default=100000, help="Upper limit for CPU-bound task (prime counting).")
    parser.add_argument("--duration", type=float, default=0.01, help="Sleep duration for IO-bound task in seconds.")
    parser.add_argument("--json", action="store_true", help="Output results in JSON format")

    args = parser.parse_args()

    gil_status = check_gil_enabled()
    results = {
        "gil_enabled": gil_status,
        "task": args.task,
        "method": args.method,
        "workers": args.workers,
        "duration": None,
        "total_primes": None
    }

    if args.task == "cpu":
        if args.method == "threading":
            duration, total = run_threading_cpu(args.n, args.workers)
            results["duration"] = duration
            results["total_primes"] = total
        elif args.method == "multiprocessing":
            duration, total = run_multiprocessing_cpu(args.n, args.workers)
            results["duration"] = duration
            results["total_primes"] = total
        else:
            print("Asyncio is not suitable for CPU-bound tasks.")
            sys.exit(1)

    elif args.task == "io":
        if args.method == "threading":
            duration = run_threading_io(args.workers, args.duration)
            results["duration"] = duration
        elif args.method == "asyncio":
            duration = asyncio.run(run_asyncio_io(args.workers, args.duration))
            results["duration"] = duration
        else:
            print("Multiprocessing is generally not used for IO-bound tasks.")
            sys.exit(1)

    if args.json:
        import json
        print(json.dumps(results))
    else:
        # Traditional output format
        print(f"GIL Enabled: {gil_status}")
        if args.task == "cpu":
            print(f"{args.method.capitalize()} CPU-bound took {results['duration']:.3f}s, total primes: {results['total_primes']}")
        else:
            print(f"{args.method.capitalize()} IO-bound took {results['duration']:.3f}s")

if __name__ == "__main__":
    main()

This script lets you choose between benchmarking CPU-bound tasks (like counting prime numbers for no good reason) or IO-bound tasks (like pretending to sleep on the job using time.sleep). Because nothing says productivity like simulating waiting.

Performance bound Benchmark (Prime Counter Edition)

This one brute-forces its way through checking for prime numbers — not because the world needs more primes, but because it's a classic way to hog the CPU. The goal isn’t mathematical insight; it’s to simulate CPU-heavy computation across threads or processes and observe how Python handles it under the GIL (or without it). Expect this to burn cycles, max out cores, and make your laptop sound like it’s trying to take off.

import matplotlib.pyplot as plt
import subprocess
import json
import pandas as pd
from typing import List, Dict
import time

def run_experiment(task_type: str, method: str, workers_list: List[int], python_version: str = "3.12", num_runs: int = 5) -> List[Dict]:
    """
    Run the benchmark experiment multiple times and collect results
    Args:
        task_type: "cpu" or "io"
        method: "threading", "multiprocessing", or "asyncio"
        workers_list: List of worker counts to test
        python_version: "3.12" (with GIL) or "3.13" (no GIL)
        num_runs: Number of times to repeat each experiment
    """
    results = []

    for workers in workers_list:
        for run in range(num_runs):
            if task_type == "cpu":
                cmd = [f"python{python_version}", "main.py", "--task", "cpu", "--method", method, 
                      "--workers", str(workers), "--n", "10000", "--json"]
            else:
                cmd = [f"python{python_version}", "main.py", "--task", "io", "--method", method,
                      "--workers", str(workers), "--duration", "0.1", "--json"]

            # Print the command being run
            print(f"\nRunning: {' '.join(cmd)}")

            output = subprocess.run(cmd, capture_output=True, text=True)

            try:
                result = json.loads(output.stdout)
                result['run'] = run  # Add run number to results
                result['python_version'] = python_version
                result['gil_enabled'] = python_version == "3.12"  # True for 3.12, False for 3.13
                results.append(result)
            except json.JSONDecodeError:
                print(f"Error parsing output: {output.stdout}")
                print(f"Stderr: {output.stderr}")

            # Small delay between runs
            time.sleep(0.1)

    return results

def create_visualizations(results: List[Dict]):
    """
    Create visualizations comparing different concurrency methods
    """
    df = pd.DataFrame(results)

    # Set style
    plt.style.use('default')

    # Create figure with subplots
    fig = plt.figure(figsize=(15, 10))

    # 1. Performance by Workers for each task type and Python version
    ax1 = plt.subplot(221)
    for task in df['task'].unique():
        task_data = df[df['task'] == task]
        for method in task_data['method'].unique():
            for version in task_data['python_version'].unique():
                data = task_data[(task_data['method'] == method) & 
                                (task_data['python_version'] == version)]
                means = data.groupby('workers')['duration'].mean()
                std = data.groupby('workers')['duration'].std()
                label = f"{task}-{method}-Python{version}"
                ax1.errorbar(means.index, means.values, yerr=std.values, 
                           label=label, marker='o')

    ax1.set_title('Performance by Number of Workers')
    ax1.set_xlabel('Number of Workers')
    ax1.set_ylabel('Duration (seconds)')
    ax1.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
    ax1.grid(True)

    # 2. Method Comparison (Box Plot)
    ax2 = plt.subplot(222)
    df.boxplot(column='duration', by=['method', 'python_version'], ax=ax2)
    ax2.set_title('Duration Distribution by Method and Python Version')
    ax2.set_ylabel('Duration (seconds)')
    plt.xticks(rotation=45)

    # 3. Speedup Analysis
    ax3 = plt.subplot(223)
    for task in df['task'].unique():
        task_data = df[df['task'] == task]
        for method in task_data['method'].unique():
            for version in task_data['python_version'].unique():
                data = task_data[(task_data['method'] == method) & 
                                (task_data['python_version'] == version)]
                baseline = data[data['workers'] == min(data['workers'])]['duration'].mean()
                speedup = data.groupby('workers')['duration'].mean().apply(lambda x: baseline / x)
                label = f"{task}-{method}-Python{version}"
                ax3.plot(speedup.index, speedup.values, label=label, marker='o')

    ax3.set_title('Speedup Analysis')
    ax3.set_xlabel('Number of Workers')
    ax3.set_ylabel('Speedup Factor')
    ax3.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
    ax3.grid(True)

    # 4. GIL Impact Analysis (Python Version Comparison)
    ax4 = plt.subplot(224)
    version_comparison = df.groupby(['method', 'python_version'])['duration'].mean().unstack()
    version_comparison.plot(kind='bar', ax=ax4)
    ax4.set_title('Average Duration by Python Version (GIL vs No GIL)')
    ax4.set_xlabel('Method')
    ax4.set_ylabel('Duration (seconds)')
    plt.xticks(rotation=45)

    plt.tight_layout()
    plt.savefig('performance_analysis.png', dpi=300, bbox_inches='tight')
    plt.close()

    # Print summary statistics
    print("\nSummary Statistics:")
    for task in df['task'].unique():
        print(f"\n{task.upper()} Tasks:")
        task_data = df[df['task'] == task]
        for method in task_data['method'].unique():
            for version in task_data['python_version'].unique():
                data = task_data[(task_data['method'] == method) & 
                                (task_data['python_version'] == version)]
                gil_status = "with GIL" if version == "3.12" else "no GIL"
                print(f"\n{method.capitalize()} (Python {version} {gil_status}):")
                print(f"Average duration: {data['duration'].mean():.3f} seconds")
                print(f"Standard deviation: {data['duration'].std():.3f} seconds")
                if task == 'cpu' and 'total_primes' in data:
                    print(f"Average primes found: {data['total_primes'].mean():.0f}")

if __name__ == "__main__":
    # Define worker configurations to test
    workers_list = [1, 2, 4, 8, 16]

    # Run experiments with Python 3.12 (with GIL)
    print("Running experiments with Python 3.12 (with GIL)...")
    results_3_12 = []
    for task_type in ["cpu", "io"]:
        for method in ["threading", "multiprocessing"] if task_type == "cpu" else ["threading", "asyncio"]:
            results_3_12.extend(run_experiment(task_type, method, workers_list, "3.12"))

    # Run experiments with Python 3.13 (no GIL)
    print("\nRunning experiments with Python 3.13 (no GIL)...")
    results_3_13 = []
    for task_type in ["cpu", "io"]:
        for method in ["threading", "multiprocessing"] if task_type == "cpu" else ["threading", "asyncio"]:
            results_3_13.extend(run_experiment(task_type, method, workers_list, "3.13"))

    # Combine all results
    all_results = results_3_12 + results_3_13

    # Create visualizations
    create_visualizations(all_results)
    print("\nVisualizations saved as 'performance_analysis.png'")

Fig- performance benchmark with accross python3.12(gil enabled) vs python3.13(gil disabled)

Performance Results: The Good, The Bad, and The Asynchronous

Here’s how our contenders fared in the performance tests.

• For CPU-Bound Tasks (the prime numbers):

  • Multiprocessing was the undisputed king. It showed near-linear speedup as we added more processes, proving that for heavy computation, bypassing the GIL by using separate processes is the most effective strategy.

  • Threading (with GIL) was, as expected, terrible. The execution time remained flat regardless of how many threads we threw at it. The GIL ensured only one thread could compute at a time, rendering the extra cores useless.

  • Threading (no GIL) was the exciting one. It showed a significant performance boost over its GIL-enabled counterpart, scaling nicely with more threads. This is the poster child for the no-GIL promise: true parallelism for CPU-bound work within a single process.

• For I/O-Bound Tasks (the waiting game):

  • Asyncio was the star performer. It handled waiting on multiple tasks with minimal overhead, making it the fastest and most efficient choice for I/O-heavy applications.

  • Threading (with or without GIL) also performed very well. Since the GIL is released during I/O operations anyway, both versions scaled effectively. This confirms that for I/O tasks, standard threading remains a simple and very viable option.

So far, so good. The no-GIL mode seems to deliver on its promise for CPU-bound work. But then we ran the corruption benchmark, and things got weird.

Corruption Benchmark (a.k.a. “Race Condition Rodeo”)

This test isn’t about speed — it’s about chaos. Here, multiple threads (or tasks, or processes, depending on your weapon of choice) increment a shared counter. It should be simple: increment a number N times. But in the absence of proper synchronization — and especially when the GIL is disabled — all bets are off. We’re trying to surface the hidden dragons: race conditions, memory corruption, and that subtle existential dread when your final count is… not quite right.

The goal is to demonstrate why the GIL was necessary — not because Python devs love locks, but because shared memory without guardrails is basically inviting entropy into your program.

import matplotlib.pyplot as plt
import subprocess
import time

def run_experiment(python_version, num_runs=5):
    """
    Run the data corruption experiment multiple times and collect results
    """
    results = []
    for _ in range(num_runs):
        if python_version == "3.13":
            cmd = ["python3.13", "data_corruption.py"]
        else:
            cmd = ["python3.12", "data_corruption.py"]

        output = subprocess.run(cmd, capture_output=True, text=True)
        lines = output.stdout.split('\n')

        # Parse the results
        actual_value = None
        expected_value = None
        execution_time = None

        for line in lines:
            if "Actual counter value:" in line:
                actual_value = int(line.split()[-1])
            elif "Expected counter value:" in line:
                expected_value = int(line.split()[-1])
            elif "Function main took" in line:
                execution_time = float(line.split()[-2])

        results.append({
            "python_version": python_version,
            "actual_value": actual_value,
            "expected_value": expected_value,
            "execution_time": execution_time,
            "lost_increments": expected_value - actual_value if expected_value and actual_value else None
        })

        # Small delay between runs
        time.sleep(0.5)

    return results

def create_visualizations(results_3_12, results_3_13):
    """
    Create various visualizations comparing Python 3.12 and 3.13 results
    """
    fig = plt.figure(figsize=(15, 10))

    # 1. Counter Values Comparison
    ax1 = plt.subplot(221)
    runs_3_12 = range(len(results_3_12))
    runs_3_13 = range(len(results_3_13))

    expected = results_3_12[0]['expected_value']
    ax1.plot(runs_3_12, [r['actual_value'] for r in results_3_12], 'g-', label='Python 3.12 (with GIL)', marker='o')
    ax1.plot(runs_3_13, [r['actual_value'] for r in results_3_13], 'r-', label='Python 3.13 (no GIL)', marker='o')
    ax1.axhline(y=expected, color='b', linestyle='--', label='Expected Value')

    ax1.set_title('Counter Values Across Runs')
    ax1.set_xlabel('Run Number')
    ax1.set_ylabel('Counter Value')
    ax1.legend()
    ax1.grid(True)

    # 2. Lost Increments Over Time
    ax2 = plt.subplot(222)
    lost_3_13 = [r['lost_increments'] for r in results_3_13]
    ax2.plot(range(len(lost_3_13)), lost_3_13, 'r-', marker='o')
    ax2.set_title('Lost Increments Over Time (Python 3.13 no GIL)')
    ax2.set_xlabel('Run Number')
    ax2.set_ylabel('Number of Lost Increments')
    ax2.grid(True)

    # 3. Execution Time Comparison
    ax3 = plt.subplot(223)
    times_3_12 = [r['execution_time'] for r in results_3_12]
    times_3_13 = [r['execution_time'] for r in results_3_13]

    ax3.bar(['Python 3.12\n(with GIL)', 'Python 3.13\n(no GIL)'],
            [sum(times_3_12)/len(times_3_12), sum(times_3_13)/len(times_3_13)],
            color=['green', 'red'])
    ax3.set_title('Average Execution Time')
    ax3.set_ylabel('Time (seconds)')
    ax3.grid(True)

    # 4. Consistency Percentage
    ax4 = plt.subplot(224)
    consistency_3_12 = sum(1 for r in results_3_12 if r['actual_value'] == r['expected_value']) / len(results_3_12) * 100
    consistency_3_13 = sum(1 for r in results_3_13 if r['actual_value'] == r['expected_value']) / len(results_3_13) * 100

    ax4.bar(['Python 3.12\n(with GIL)', 'Python 3.13\n(no GIL)'], 
            [consistency_3_12, consistency_3_13],
            color=['green', 'red'])
    ax4.set_title('Consistency Percentage')
    ax4.set_ylabel('Percentage of Consistent Results')
    ax4.set_ylim(0, 100)
    ax4.grid(True)

    plt.tight_layout()
    plt.savefig('corruption_analysis.png')
    plt.close()

    # Print summary statistics
    print("\nSummary Statistics:")
    print("\nPython 3.12 (with GIL):")
    print(f"Average execution time: {sum(times_3_12)/len(times_3_12):.3f} seconds")
    print(f"Consistency rate: {consistency_3_12:.1f}%")

    print("\nPython 3.13 (no GIL):")
    print(f"Average execution time: {sum(times_3_13)/len(times_3_13):.3f} seconds")
    print(f"Consistency rate: {consistency_3_13:.1f}%")
    print(f"Average lost increments: {sum(lost_3_13)/len(lost_3_13):.0f}")

if __name__ == "__main__":
    # Run experiments
    print("Running experiments with Python 3.12...")
    results_3_12 = run_experiment("3.12")

    print("Running experiments with Python 3.13...")
    results_3_13 = run_experiment("3.13")

    # Create visualizations
    create_visualizations(results_3_12, results_3_13)
    print("\nVisualizations saved as 'corruption_analysis.png'")

fig: bench-marking memory corruption

Corruption Results: A Slow-Motion Car Crash

This is where my assumptions were shattered. I expected the no-GIL threads to be fast but wrong. I was not prepared for them to be slow and wrong.

Let's dissect this beautiful disaster:

1. Consistency: As predicted, Python 3.12 (with the GIL) was 100% correct every time. Python 3.13 (no GIL), without any manual locks, produced wildly incorrect results, losing tens of thousands of increments to race conditions. No surprises there.

2. Execution Time: This was the shocker. The no-GIL version was significantly slower than the GIL version. It not only failed to produce the right answer, it took its sweet time doing it.

Why in the World Was It Slower?

This counter-intuitive result gets to the heart of concurrency. Removing the GIL isn't free. It's replaced by other, more granular mechanisms like atomic operations to keep Python's internals safe. And our specific test—multiple threads hammering the exact same variable—is the absolute worst-case scenario for this.

It created extreme memory contention.

Think of it this way: The GIL is like a strict librarian who only lets one person at a time into a room to get a book. It's slow, but it's orderly. The no-GIL mode fires the librarian and lets everyone rush in at once. But since they all want the same book, they just trample each other in a chaotic pile-up at the bookshelf. The CPU's memory system has to work overtime to manage this riot, and the end result is that everything takes longer than the orderly queue did.

Our test created a digital riot. The overhead of managing the chaos was greater than the benefit of parallelism.

The Grand, Unfulfilling Conclusion: So What Did We Learn?

After all this, the key takeaway is that there is no magic bullet. Anyone who tells you "always use async" or "the no-GIL mode fixes everything" is, frankly, full of it.

The reality requires you to think. Here's your cheat sheet:

1. For CPU-Bound Work: Multiprocessing is still your safest bet for raw, scalable power. Use the no-GIL mode only when your threads can work on different chunks of data with low contention. Benchmark it first, or you might end up in a slow-motion riot like I did.

2. For I/O-Bound Work: Asyncio is the modern, highly efficient champion. But standard threading (with the GIL) remains a perfectly simple and effective choice for less complex I/O tasks.

The no-GIL mode is not a "go faster" button. It's a powerful, expert-level tool that unlocks true parallelism, but it demands that you understand and manage the perils of concurrent memory access. For most, the GIL isn't a prison; it's a guardrail that keeps you from driving off a cliff. Sometimes, that's exactly what you need.

Let’s be honest: writing blogs is exhausting, and reading AI-generated "thought leadership" on LinkedIn is even worse. I didn’t want to contribute to the noise — but here i am pretending to create something actually useful. And maybe, just maybe, save myself from typing the same “Here’s how you use a decorator” explanation for the 20th time.

So I built two things. And now, we’re pitting them against each other in a cage match of code and questionable sanity:

1. My Hand-Rolled Agent: A custom-built Python script, fueled by LLMs and sheer willpower, designed to take a blog title, some rough notes, and tags, and spit out a semi-coherent Hashnode post. I even built a whole UI for this monstrosity.

2. MCP (Model Context Protocol): I was going to implement it myself. But i found a opensource implementation of it done very recently by someone who had an awesome portfolio site. Same like mine but on steroids.

For those who don’t know — Hashnode is basically like Medium, but built for developers and, honestly, way better. Custom domains, Markdown, no paywalls, and an actual API? Yeah, Medium could never. This blog site made with Hashnode.

1. 💀 My Hand-Rolled Agent

A custom-built Python script, fueled by LLMs and sheer willpower, designed to take a blog title, some rough notes, and tags, and spit out a semi-coherent Hashnode post.

I even built a whole UI for this monstrosity. It’s rough around the edges, but it’s mine(claude). It accepts your blog title, tags, and raw chaotic notes like:

Then it wrangles that input into something that won't get you cyberbullied on Twitter. The backend? FastAPI. The LLM? Gemini. The deploy? Docker Compose. The pain? Immeasurable.

Feast your eyes on the screenshots:

This setup gave me a decent amount of control, and honestly, it felt pretty great to spin up something from scratch. The full thing — frontend, backend, all duct-taped together — is open source at agent-hashnode. It’s fast, it’s messy, and it works.

2. 🤖 The Hashnode MCP (Model Context Protocol)

Okay, so here’s where the future slaps you in the face.

I had barely dipped my toes into the whole MCP ecosystem, and suddenly I'm staring into a rabbit hole of possibility. I was going to build my own implementation. I really was. But then I stumbled across an very recent opensource contribution. Shoutout to sbmagar13 for building the actual mcp server.

Now, what is this "MCP" thing?

Imagine your dumb little script from earlier. Then imagine you could talk to it. Like, literally say:

"Hey, create a blog post about FastAPI and Gemini, tag it with 'AI', 'Python', 'FastAPI', and use these notes to flesh it out."

And it does that. Not because you hardcoded anything, but because the LLM running behind it actually understands the task and knows how to call different microservices to get it done.

I spun up a TUI-based MCP client with Cursor (yes, that Cursor), and here's what it looks like:

With just natural language, I can now:

• 🗂 Create folders and files

• 📄 Generate blogs (duh)

• 📝 Edit existing blog posts

• 🧠 Chain tasks like "create blog → upload to Hashnode → tweet it"

• 🧰 Add more servers to automate literally everything

Everything’s modular. Everything’s extendable. I could hook it up to an email server, a Terraform infra manager, even a coffee machine if I wanted. I’m basically living in 2035 now.

Here’s what it looks like when I ask it to publish a blog using the Hashnode MCP server:

No manual API calls. No YAML sacrifices to the CI/CD gods. Just vibes and language. Look how good of a poem it wrote about my friend paarit.

The full implementation of the Hashnode MCP is at sbmagar13/hashnode-mcp-server. Seriously, check it out. Kudos to the developer

🧠 Final Thoughts

I started this project thinking I’d hack something together, automate a few boring blog posts, and call it a day. But what I discovered was a fundamental shift in how we build tools.

Originally, I just wanted to continue my Advanced Python blog series. I had learned some cool new tricks and figured it was time to share. But the idea of sitting down to write all those blogs — with proper code snippets, formatting, explanations — sounded exhausting. And of course i would be using LLM anyway.

So I thought, “Why not automate the whole thing?” That’s when I stumbled across MCP. I didn’t know much, but figured I’d learn it and use it to generate and publish blogs. Of course, being me, I ended up starting coding immediately and buildt my own agent from scratch.

...which, hilariously, wasn’t MCP at all.

But then I looked into the real MCP — and man. It looked extremely promising. On my journey to learn MCP and automate my blogs. Within six hours I had built three repos, written five blogs, and even spun up my own custom MCP client from scratch. I don’t even know what code has been written at this point(thank you cursor). It just happened.

The scale at which we can build things now is unprecedented. This isn’t about speed anymore — it’s about lightspeed(okay this is lame).

The classical method? It works. You build parts, wire them together, and maybe—just maybe—you get something that saves you 10 minutes a week.

But MCP? It’s not just code. It’s conversation. It’s building with your tool, not just for it. It’s not magic — it’s just damn good architecture powered by LLMs that actually understand how to collaborate with APIs.

If you’re a developer, go try this. Stop gluing together scripts. Start treating your tools like collaborators.

The world is changing. Fast.

And honestly?

Bye bye coders. Hahahaha.

References

Code

1. MCP Terminal Client

2. Custom Hasnode AI agent

3. Hashnode MCP server

Blogs Generated

1. Python descriptor protocol

2. Python closure

3. Python ___hash___ dunder

4. Bench-marking parallelism with GIL enabled&disabled

5. Paarit Lonely Poetry

6. blog generated by very random prompt

This blog is generated as a usecase of this other blog(Automating blog creation with own agent vs mcp) that helped me automate the writing of this blog.

Alright, settle in, folks. Today we're diving into Python descriptors. Not because some LinkedIn guru told me it's the "next big thing" (eye roll), but because they're genuinely useful for building reusable, elegant, and frankly, kinda badass attribute logic. If you're tired of copy-pasting validation code across your classes, or you're just curious about the magic behind @property, then buckle up. We're going deep.

What in the Holy Hell is a Descriptor?

In essence, a descriptor is a class that manages attribute access. Woah, slow down! Think of it as a gatekeeper for your object's attributes. Instead of directly accessing the attribute, you go through the descriptor, which can then perform extra logic like validation, lazy loading, or even tracking changes.

Descriptors are defined by implementing one or more of the following special methods (the descriptor protocol):

• __get__(self, instance, owner): Called when the descriptor's attribute value is accessed.

• __set__(self, instance, value): Called when the descriptor's attribute value is assigned.

• __delete__(self, instance): Called when the descriptor's attribute is deleted.

• __set_name__(self, owner, name): Called when the owning class is created, giving the descriptor a chance to know its attribute name. (Python 3.6+)

These methods let you intercept attribute access, assignment, and deletion, giving you fine-grained control over how your objects behave.

The @property Decorator: Descriptors in Disguise

You've probably used @property before, right? It's that handy decorator that lets you define getter, setter, and deleter methods for an attribute. Guess what? It's built on descriptors!

Let's break it down:

class Circle:
    def __init__(self, radius):
        self._radius = radius

    @property
    def radius(self):
        """I'm the 'radius' property."""
        print("Getting radius")
        return self._radius

    @radius.setter
    def radius(self, value):
        if value <= 0:
            raise ValueError("Radius must be positive")
        print("Setting radius")
        self._radius = value

    @radius.deleter
    def radius(self):
        print("Deleting radius")
        del self._radius

Under the hood, @property creates a property object, which is a descriptor. When you access circle.radius, Python calls the __get__ method of the property object, which in turn calls your getter method. The same goes for setting and deleting the attribute.

@radius.setter and @radius.deleter are just syntactic sugar to create new property objects with the setter and deleter methods attached, respectively. They essentially replace the original property object associated with radius.

Roll Your Own: Custom Descriptor Examples

Okay, enough theory. Let's get our hands dirty with some custom descriptors.

Example 1: Validating Attributes

Tired of writing the same validation logic over and over? A descriptor can help.

class ValidatedString:
    def __init__(self, min_length=0, max_length=None):
        self.min_length = min_length
        self.max_length = max_length
        self.name = None # Assigned by __set_name__

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, instance, owner):
        if instance is None:
            return self
        return instance.__dict__[self.name]

    def __set__(self, instance, value):
        if not isinstance(value, str):
            raise TypeError(f"{self.name} must be a string")
        if len(value) < self.min_length:
            raise ValueError(f"{self.name} must be at least {self.min_length} characters long")
        if self.max_length is not None and len(value) > self.max_length:
            raise ValueError(f"{self.name} must be at most {self.max_length} characters long")
        instance.__dict__[self.name] = value


class User:
    name = ValidatedString(min_length=3, max_length=50)
    email = ValidatedString()

    def __init__(self, name, email):
        self.name = name
        self.email = email

Now, whenever you try to assign an invalid value to user.name or user.email, the ValidatedString descriptor will raise an exception. Boom. Reusable validation.

Example 2: Lazy Loading

Want to defer loading an expensive attribute until it's actually needed? Descriptors to the rescue!

import time

class LazyLoad:
    def __init__(self, func):
        self.func = func
        self.name = None # Assigned by __set_name__

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, instance, owner):
        if instance is None:
            return self
        print(f"Loading {self.name}...")
        value = self.func(instance)
        instance.__dict__[self.name] = value  # Cache the result
        return value


class DataProcessor:
    @LazyLoad
    def large_dataset(self):
        # Simulate loading a large dataset
        print("Actually loading...")
        time.sleep(2)
        return list(range(1000000))

    def __init__(self):
        pass

The first time you access data_processor.large_dataset, the LazyLoad descriptor will call the large_dataset method, load the data, and cache it in the instance's __dict__. Subsequent accesses will retrieve the cached value directly, avoiding the expensive loading process. Efficiency!

Real-World Applications: Beyond the Hype

Descriptors aren't just academic exercises. They're used in real-world applications all the time.

• ORMs (Object-Relational Mappers): ORMs like SQLAlchemy use descriptors to map database columns to object attributes, handle data validation, and track changes.

• Validation Libraries: Libraries like attrs and marshmallow leverage descriptors for defining and validating object attributes.

• Frameworks: Django uses descriptors in its model fields to handle data conversion and validation.

Basically, if you're working on a complex Python project, chances are you're already using descriptors, even if you don't realize it.

Common Pitfalls and Best Practices

Descriptors can be powerful, but they can also be tricky. Here are a few things to keep in mind:

• Descriptor Types: There are two main types of descriptors: data descriptors (with both __get__ and __set__) and non-data descriptors (with only __get__). Data descriptors take precedence over instance attributes, while non-data descriptors are shadowed by instance attributes. Understand the difference!

• __set_name__: Don't forget to use __set_name__ (Python 3.6+) to get the attribute name. It makes your descriptors more reusable and less prone to errors.

• Avoid Infinite Recursion: Be careful when accessing or setting attributes within your descriptor methods. You can easily create infinite recursion if you're not careful. Use the instance's __dict__ directly to avoid triggering the descriptor again.

• Keep it Simple: Don't overcomplicate your descriptors. If you find yourself writing a ton of code, consider breaking it down into smaller, more manageable functions or classes.

Conclusion: Embrace the Descriptor

Python descriptors are a powerful tool for building reusable and maintainable code. They might seem a bit intimidating at first, but once you understand the basics, you'll be able to write more elegant and efficient Python code. So, go forth and embrace the descriptor! And remember, don't let the LinkedIn influencers tell you what to do. Think for yourself, write good code, and stay slightly unhinged. You'll go far.

This blog is generated as a usecase of this other blog(Automating blog creation with own agent vs mcp) that helped me automate the writing of this blog.

Alright, buckle up buttercups. Today we're diving into the murky, sometimes terrifying, world of hashable objects in Python. Forget those LinkedIn influencers droning on about "synergy" and "disruption." We're talking about the actual nuts and bolts of making your Python code not just work, but work efficiently and predictably. And yes, I'm going to make fun of something along the way. Probably many things.

What in the Holy Guido is a Hashable Object?

Simply put, a hashable object is one that has a hash value that never changes during its lifetime. This hash value is an integer, and it's used by Python to quickly look up objects in dictionaries and sets. Think of it like a social security number for your data. Once assigned, it's (supposed to be) permanent.

More formally, an object is hashable if it has:

• A __hash__() method which returns an integer.

• An __eq__() method to compare it to other objects for equality.

The crucial point? If two objects are equal (i.e., a == b is True), then their hash values must be the same (i.e., hash(a) == hash(b)). Conversely, if their hash values are different, they should be unequal. (Collisions happen, but minimizing them is the goal.)

"TypeError: unhashable type: 'list'" - A Tale of Woe

Ever seen this error? It usually pops up when you try to use a list as a key in a dictionary or as an element in a set. Let's see it in action:

my_list = [1, 2, 3]
try:
    my_dict = {my_list: "value"}
except TypeError as e:
    print(f"Caught the expected error: {e}")

try:
    my_set = {my_list}
except TypeError as e:
    print(f"Caught the expected error: {e}")

Why does this happen? Because lists are mutable. You can change them after they're created. If you could use a list as a dictionary key, and then change the list, the dictionary would no longer be able to find the value associated with that key. Chaos would reign. Dogs and cats living together! Mass hysteria!

The __eq__ Connection: They're Married, Deal With It

The __hash__() and __eq__() methods are inextricably linked. If you define __eq__() for your custom class, you must also define __hash__(). And, as I mentioned earlier, if two objects are equal according to __eq__(), their hash values must be the same.

If you define __eq__ but not __hash__, Python will implicitly set __hash__ = None, and your object will be unhashable. This is a safety mechanism to prevent you from accidentally breaking the hash table invariants. Python is looking out for you, even if it feels like a passive-aggressive intervention.

Leveraging Hashability: Dictionaries and Sets, Baby!

The primary benefit of hashable objects is their ability to be used as keys in dictionaries and elements in sets. These data structures rely on hashing for fast lookups and membership tests. If you're doing a lot of searching or checking for duplicates, using dictionaries or sets with hashable objects can dramatically improve your code's performance.

Consider this highly contrived example:

# A list of names
names = ["Alice", "Bob", "Charlie", "Alice", "David", "Bob"]

# Using a set to find unique names (efficiently!)
unique_names = set(names)
print(f"Unique names: {unique_names}")

# Checking if a name is in the set (also efficient!)
if "Alice" in unique_names:
    print("Alice is unique-ish.")

The set() constructor automatically removes duplicates because it relies on hashing. And checking for membership in a set (using the in operator) is much faster than iterating through a list.

Mutability and Hashability: A Thorny Relationship

Generally, mutable objects (like lists and dictionaries) are not hashable, while immutable objects (like tuples, strings, and numbers) are hashable. This is because the hash value of an object must remain constant throughout its lifetime. If you could change a mutable object after it's been used as a dictionary key, the dictionary would become corrupted.

However, there are exceptions! You can create custom mutable objects that are hashable, but you need to be very careful. The key is to ensure that the hash value is based on attributes that never change after the object is created. This is tricky, and I generally advise against it unless you have a very good reason.

dataclasses to the Rescue (Sometimes)

Python's dataclasses module can automatically generate __hash__() and __eq__() methods for you, based on the fields you define in your data class. By default, a dataclass is hashable if all of its fields are hashable. You can also explicitly specify that a dataclass should be frozen (immutable) using the @dataclass(frozen=True) decorator.

Here's an example:

from dataclasses import dataclass

@dataclass(frozen=True)
class Point:
    x: int
    y: int

p1 = Point(1, 2)
p2 = Point(1, 2)

print(f"p1 == p2: {p1 == p2}")
print(f"hash(p1) == hash(p2): {hash(p1) == hash(p2)}")

my_dict = {p1: "origin"} # This now works!

By setting frozen=True, we've made the Point class immutable and hashable. This is a great way to create simple, data-centric classes that can be used as dictionary keys or set elements.

Conclusion: Embrace the Hash, Fear the Mutability

Hashable objects are fundamental to Python's data structures and performance. Understanding the relationship between __hash__(), __eq__(), and mutability is crucial for writing robust and efficient code.

So, go forth and hash! But remember, with great power comes great responsibility (and the potential for really weird bugs if you screw it up). And for the love of all that is holy, stop listening to those LinkedIn influencers. They're probably just trying to sell you something. Now, if you'll excuse me, I need to go yell at a cloud.

This blog is generated as a usecase of this other blog(Automating blog creation with own agent vs mcp) that helped me automate the writing of this blog.

Alright, buckle up buttercups. Today we're diving into Python closures. Not the LinkedIn-influencer-approved "OMG closures are so elegant" kind. We're talking about the real closures, the ones that can bite you in the ass if you're not paying attention. Think of this as the anti-bullshit guide to closures. Because let's be honest, half the "Python experts" out there just parrot the same definition without actually understanding the underlying mechanisms. Let's fix that.

Tags: Python, Closures, Decorators, Lexical Scoping, Python Internals, Back end Engineering

What the Hell Is a Closure Anyway?

Okay, so the textbook definition goes something like this: A closure is a function object that remembers values in enclosing scopes even if they are not present in memory. Sounds like consultant speak, right? Let's break it down.

Essentially, a closure is a function bundled together with its surrounding state (its lexical environment). This environment consists of any free variables (variables used but not defined within the function itself) that were in scope when the closure was created.

Think of it like this: you're packing a lunchbox. The lunchbox is your function. The sandwich, apple, and juice box are the variables from the surrounding scope. Even after you leave the kitchen (the original scope), your lunchbox (the function) still has access to the sandwich, apple, and juice box. Got it? Good.

Scoping: The Foundation of All Evil (and Closures)

Before we go any further, let's revisit scoping in Python. Python uses lexical scoping, also known as static scoping. This means that a variable's scope is determined by its position in the source code.

We have four main types of scopes:

• Local: Inside a function.

• Enclosing: In the scope of an enclosing function (where closures come into play!).

• Global: At the top level of a module.

• Built-in: Predefined names in Python (like print, len, etc.).

Python follows the LEGB rule: Local -> Enclosing -> Global -> Built-in. When you try to access a variable, Python searches these scopes in that order until it finds the variable.

x = 10 # Global scope

def outer_function():
    y = 20 # Enclosing scope

    def inner_function():
        z = 30 # Local scope
        print(x, y, z) # Accessing variables from all scopes

    inner_function()

outer_function() # Output: 10 20 30

In the above example, inner_function has access to x (global), y (enclosing), and z (local). This is the foundation for closures.

Closures in Action: Let's Get Practical

Here's where the magic happens. Let's create a function that returns another function, capturing a variable from its enclosing scope:

def multiplier(n):
    def inner(x):
        return x * n
    return inner

double = multiplier(2)
triple = multiplier(3)

print(double(5)) # Output: 10
print(triple(5)) # Output: 15

See what's happening? The multiplier function returns the inner function. Critically, inner "remembers" the value of n from the multiplier function's scope, even after multiplier has finished executing. That's a closure, baby! double and triple are closures, each with their own captured value of n.

Closures and Decorators: A Match Made in Python Heaven (or Hell, Depending on Your Debugging Skills)

Closures are the backbone of Python decorators. Decorators are syntactic sugar for wrapping functions with other functions. Let's see how:

def my_decorator(func):
    def wrapper():
        print("Before calling the function.")
        func()
        print("After calling the function.")
    return wrapper

@my_decorator
def say_hello():
    print("Hello!")

say_hello()
# Output:
# Before calling the function.
# Hello!
# After calling the function.

Under the hood, the @my_decorator syntax is equivalent to:

say_hello = my_decorator(say_hello)

my_decorator returns the wrapper function, which is a closure that has access to the original func. This allows you to add functionality to a function without modifying its original code. Pretty neat, huh?

The Closure Trap: Beware the Late Binding!

This is where things get tricky. Let's say you want to create a list of functions, each multiplying by a different number:

def create_multipliers():
    multipliers = []
    for i in range(5):
        multipliers.append(lambda x: x * i)
    return multipliers

multipliers = create_multipliers()

for multiplier in multipliers:
    print(multiplier(2))

What do you expect the output to be? 0, 2, 4, 6, 8? Nope! You'll get 8, 8, 8, 8, 8. Why? Because the lambda functions are closures that capture the variable i, not its value at the time of creation. By the time you call the functions, the loop has finished, and i is equal to 4.

This is the late binding or closure trap. To fix it, you need to force the value of i to be bound at the time the function is created:

def create_multipliers_fixed():
    multipliers = []
    for i in range(5):
        multipliers.append(lambda x, i=i: x * i) # Bind i to the argument's default value
    return multipliers

multipliers = create_multipliers_fixed()

for multiplier in multipliers:
    print(multiplier(2))
#Output:
#0
#2
#4
#6
#8

By using a default argument, we're creating a new variable i within the scope of each lambda function, effectively capturing its value at the time of creation. Boom. Problem solved.

Closures: Python vs. The World

Closures are a common feature in many programming languages, but their implementation and behavior can vary. In languages like JavaScript, closures are ubiquitous and often used for event handling and asynchronous programming. In languages like C++, closures are often implemented using lambda expressions and function objects.

Python's closures are relatively straightforward, but the late binding issue can be a source of confusion for beginners. Understanding how closures work under the hood is crucial for writing correct and efficient Python code.

Conclusion: Closures Are Your Friends (Mostly)

Closures are a powerful tool in Python, enabling you to create flexible and reusable code. They're essential for understanding decorators and other advanced Python concepts. Just remember to watch out for the late binding trap, and you'll be golden.

Now go forth and conquer the world with your newfound closure knowledge! And for god's sake, stop reading LinkedIn and start writing some actual code. You'll thank me later.

Reflection

Reflection is a powerful feature in programming that allows a program to inspect and interact with its own structure and behavior at runtime. In simpler terms, reflection allows your code to examine itself. It can look at the types, attributes, and methods of objects and classes dynamically, and even modify them.

Some common examples of reflection used in Python are setattr, getattr, callable, issubclass, and inspect.getsource.

Type annotations

Type annotations have been part of Python for some time now. However, since Python is dynamically typed, these annotations aren't enforced by the interpreter but are used by static type checkers like mypy and other LSPs. I believe that combining these type annotations with reflection can be a powerful tool to create your data validators. In fact, Pydantic is developed based on these concepts.

def sum(no1: int, no2: int)-> int:
    return no1 + no2 

sum.__annotations__
# Output - {'no1': int, 'no2': int, 'return': int}

class Vehicle:
    mileage: float
    top_speed: float
    odometer_reading: int

Vehicle.__annotations__
# Output - {'mileage': float, 'top_speed': float, 'odometer_reading': int}

The above example explains how type annotations are stored as dunder attributes in a python class.

Dataclass

from dataclasses import dataclass

@dataclass
class Vehicle:
    mileage: float
    top_speed: float
    odometer_reading: int

vehicle = Vehicle(42, 120, 45000)
print(vehicle.__str__())
# Output: Vehicle(mileage=42, top_speed=120, odometer_reading=45000)
vehicle.__dict__
# Output: {'mileage': 42, 'top_speed': 120, 'odometer_reading': 45000}

The above code shows how a dataclass works. We can see that various special methods are generated, and a constructor is created that takes the class annotations as parameters and initializes them as instance variables.

This is not a built-in Python feature; all this work is done by the dataclass decorator. Now, we will create our own minimal dataclass decorator that performs the same tasks.

Implementation

Now let's see what happens when we don't use any dataclass decorator.

from dataclasses import dataclass

class Vehicle:
    mileage: float
    top_speed: float
    odometer_reading: int

vehicle = Vehicle(42, 120, 45000)
# Output(error): TypeError: Vehicle() takes no arguments

Without the dataclass decorator, special methods like __init__ are not implemented. Using reflection and annotations, we will create these special methods ourselves.

For this we will first create a decorator named datakilas.

def datakilas(cls):
    annotations = cls.__annotations__
    print(f"{annotations=}")
    # logic to dynamically assign __init__ method into cls
    return cls

@datakilas
class Vehicle:
    mileage: float
    top_speed: float

# Output: annotations={'mileage': <class 'float'>, 'top_speed': <class 'float'>}

Now that we know our next task, let's dynamically create the constructor and representational (__str__) functions for the decorated class.

def datakilas(cls):
    annotations = cls.__annotations__
    print(f"{annotations=}")

    # Define the __init__ method for the class
    def init_func(self, **kwargs):
        for key in annotations:
            setattr(self, key, kwargs.get(key, None))

    # Define the __str__ method
    def str_func(self):
        # Create a string representation of the class with its attributes
        attr_str = ', '.join(f'{key}={getattr(self, key)}' for key in annotations)
        return f'{cls.__name__}({attr_str})'

    # Add the __init__ and __str__ method to the class attributes
    cls.__init__ = init_func
    cls.__str__ = str_func

    return cls

In the above program, we dynamically create the __init__ and __str__ dunders using the class annotations. Now we're using the new datakilas decorator, we can easily create a simple dataclass.

@datakilas
class Vehicle:
    mileage: float
    top_speed: float
vehicle = Vehicle(mileage=15, top_speed=200)

print(vehicle)
# Output: Vehicle(mileage=15, top_speed=200)

print(vehicle.__dict__)
# Output: {'mileage': 15, 'top_speed': 200}

The current implementation expects keyword arguments in class constructors. To understand it better, you can update the implementation to accept non-keyword arguments as well.

Conclusion

This way, we can easily create a basic version of a complex feature like dataclass using Python's simple reflection abilities. Although we didn't explicitly use reflection-specific tools, we used Python's powerful language features to update a class's behavior at runtime and dynamically add constructors and other methods to it. I think this serves as a perfect example of the power of reflection in Python, showing how easy and sophisticated it is to write complex programming logic in Python.