স্কেলিং NLP সিস্টেম

১ user এ API কাজ করছে। ১০০ user — ঠিক আছে। ১ million daily request? — server crash, latency 10 second, GPU বিল $10K/month। NLP system scale করা মানে শুধু big server না — smart architecture। Batching, caching, model optimization, load balancing — সব কিছু combine করে millions of request handle।

Scaling strategy: (1) Horizontal scaling — multiple replica + load balancer, (2) Vertical scaling — bigger GPU/CPU, (3) Dynamic batching — multiple request একসাথে process (Triton, vLLM), (4) Model optimization — quantization (INT8), distillation, ONNX, TensorRT, (5) Caching — frequent query এর response cache (Redis), (6) Async queue — long task background worker (Celery, RQ)।

ভাবুন একটা restaurant — peak hour এ চলছে। Solution: (1) আরো branch খুলুন (horizontal), (2) বড় kitchen বানান (vertical), (3) একই order দশজনের একসাথে রান্না করুন (batching), (4) chef এর recipe efficient করুন (model optimization), (5) popular dish আগে থেকে রেডি (cache), (6) catering order background এ ready (async queue)।

• OpenAI — vLLM/TensorRT-LLM দিয়ে massive throughput।
• Triton Inference Server — NVIDIA এর batching engine।
• ChatGPT — KV cache + speculative decoding।
• Redis caching common query এ।
• Kubernetes HPA (Horizontal Pod Autoscaler)।

from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel
from transformers import pipeline
import asyncio
import hashlib
import time
from collections import defaultdict

app = FastAPI()
classifier = pipeline("sentiment-analysis")

cache = {}
batch_queue: list = []
batch_lock = asyncio.Lock()
BATCH_SIZE = 8
BATCH_WAIT_MS = 50

class Req(BaseModel):
    text: str

def cache_key(text: str) -> str:
    return hashlib.md5(text.encode()).hexdigest()

async def process_batch():
    async with batch_lock:
        if not batch_queue:
            return
        items = batch_queue[:BATCH_SIZE]
        del batch_queue[:BATCH_SIZE]

    texts = [it["text"] for it in items]
    results = classifier(texts)
    for it, res in zip(items, results):
        cache[cache_key(it["text"])] = res
        it["future"].set_result(res)

async def batch_worker():
    while True:
        await asyncio.sleep(BATCH_WAIT_MS / 1000)
        if batch_queue:
            await process_batch()

@app.on_event("startup")
async def startup():
    asyncio.create_task(batch_worker())

@app.post("/predict")
async def predict(req: Req):
    key = cache_key(req.text)
    if key in cache:
        return {"cached": True, "result": cache[key]}

    fut = asyncio.get_event_loop().create_future()
    async with batch_lock:
        batch_queue.append({"text": req.text, "future": fut})

    result = await fut
    return {"cached": False, "result": result}

• Cache invalidation strategy নেই — stale data।
• Batch size খুব বড় — latency বাড়ে।
• Async function এ sync model call — event loop block।
• Auto-scaling threshold ভুল — over-provision/under-provision।
• Monitoring ছাড়া scale — কোথায় bottleneck জানা যায় না।

1. Locust/k6 দিয়ে load test, 1000 RPS handle করুন।
2. ONNX এ model convert করে latency compare।
3. Redis cache integrate করুন।
4. Kubernetes HPA setup করে auto-scale demonstrate।

• Scaling = horizontal + vertical + smart optimization।
• Batching, caching, quantization — throughput multiplier।
• Async queue দিয়ে long task offload।
• K8s + autoscaler দিয়ে demand-based replica।
• Architecture decision data driven (profile first)।