From social signals to strategic intelligence

An internal account-management team needed a faster way to understand how audiences were responding to social content. The desired workflow sounded simple: paste a batch of YouTube, Instagram, or TikTok post URLs, collect the comments, classify each one as positive, neutral, or negative, and download the results.

The operating reality made that workflow an architecture problem.

The initial audience was fewer than ten users, so a permanently overprovisioned platform would be wasteful. Usage would be bursty: long quiet periods followed by large URL batches. A single job could involve pagination, platform rate limits, thousands of comments, and many minutes of processing. The team accepted that results did not need to be immediate, but they did need to be traceable and available when users returned.

The brief therefore combined four tensions:

• deliver a low-cost MVP in two to three weeks;
• keep the interface responsive during long-running work;
• create sentiment outputs that people could inspect and challenge;
• avoid a throwaway prototype if adoption accelerated.

The most important product decision was to treat asynchronous completion as part of the user experience, not as invisible backend machinery. Users would submit a job, receive an identifier immediately, leave the processing to the platform, and return to a durable execution history.

I began with the job to be done rather than a preferred framework:

When an account manager needs to understand audience response to one or more social posts, they want to submit the URLs once and return to a completed analysis, so they can identify risks and communicate performance without manually reading every comment.

That statement established the minimum complete learning loop.

1. An authenticated user submits one or more supported post URLs.
2. The product validates the request and returns a job ID immediately.
3. Background workers collect comments and run sentiment analysis.
4. The platform stores both job state and comment-level results.
5. The user revisits the execution, reviews aggregates and individual comments, and downloads a CSV.
6. Incorrect classifications can be flagged so model quality improves from real use.

The MVP deliberately stops there. It does not begin with real-time streaming, automated replies, broad social listening, or an expensive large-language-model call for every comment. The first release exists to test whether faster, repeatable comment analysis changes decisions enough to justify the next investment.

The platform separates fast request intake from slow, failure-prone background work.

FastAPI acts as the authenticated boundary. It validates a URL batch, creates a unique job ID, places the job on a Redis-backed queue, and returns 202 Accepted. Celery workers running on AWS ECS Fargate claim the work independently, call platform adapters, handle pagination and retries, run the sentiment engine, and persist structured results in PostgreSQL.

Redis serves three carefully bounded MVP roles: message broker, short-lived job state, and cache. PostgreSQL remains the durable system of record for jobs, posts, comments, scores, confidence, and user feedback. When a user requests an export, the API creates a CSV in a private S3 bucket and returns a short-lived download URL, keeping large file transfer away from the API process.

This design accepts a slower time-to-result in exchange for a faster and more reliable interaction. It also lets intake capacity, worker capacity, reporting reads, and storage evolve independently.

FastAPI at the request boundary

FastAPI was chosen for an asynchronous ASGI request model and explicit Pydantic validation. Those capabilities fit bursty, I/O-heavy intake and provide a clear contract before work enters the queue. The API is intentionally kept thin: authorize, validate, create state, enqueue, and retrieve results.

Celery for durable background work

Celery provides the execution model for long-running comment collection and analysis. Built-in retry policies support exponential backoff for transient network and upstream API failures. Late task acknowledgement means a worker crash does not silently discard work; another worker can retry the unacknowledged task.

The trade-off is additional operational state. Jobs need idempotency, visible failure reasons, bounded retries, and clear partial-completion rules. Those are better problems than tying a many-minute workload to an HTTP request.

Fargate instead of Lambda or self-managed servers

Some jobs could exceed serverless-function execution limits, making a function-only design a poor fit. ECS Fargate supports long-running containers without requiring the team to patch and manage virtual machines. It also suits a workload where worker count should rise with queue pressure and fall during inactivity.

The trade-off is managed-container cost and orchestration complexity. For the MVP, that cost buys a cleaner route from a small deployment to horizontal worker scaling.

PostgreSQL instead of a document store

The core data is relational: users own jobs; jobs contain posts; posts contain comments; comments have model outputs and feedback. PostgreSQL supports that structure, the joins required for reporting, and row-level security as a second authorization boundary.

The longer-term design adds read replicas for analytical traffic and time-based partitioning when comment volume makes whole-table scans inefficient. Those capabilities are staged responses to measured pressure, not day-one complexity.

Redis as a deliberate MVP simplification

Using Redis for the queue, job state, and cache reduces the number of moving parts in the first release. A cache-aside policy prevents repeated retrieval of recently processed URLs, with freshness-based expiry rather than one arbitrary lifetime for every post.

The trade-off is concentration. Queue durability, eviction policy, memory pressure, and the separation between transient state and durable results must be explicit. PostgreSQL remains authoritative; Redis is an accelerator and coordination layer.

The model strategy starts with the cheapest credible baseline and creates an evidence-based path to greater accuracy.

For the MVP, VADER provides fast, local sentiment scoring tuned to informal social language. Each comment produces three fields: a score from -1.0 to +1.0, a positive/neutral/negative label, and a confidence value. The confidence field is not cosmetic. It controls review priority and creates a route for comparing inexpensive baseline analysis with more capable models.

The planned second tier sends only ambiguous or low-confidence comments to an LLM. This keeps easy cases local while reserving paid contextual reasoning for sarcasm, mixed sentiment, and other hard cases. The escalation threshold should be calibrated from labelled data rather than fixed by intuition.

That approach trades theoretical maximum accuracy for controllable unit economics. It also keeps the provider boundary replaceable: model choice can change without rewriting job processing, storage, or the user workflow.

A sentiment label becomes useful only when people agree what the labels mean and can see where the model fails.

The validation plan begins with two reviewers independently labelling a representative sample. Cohen's kappa checks whether human agreement is strong enough for that sample to serve as ground truth. Precision, recall, and F1 are then measured by class so a high average cannot hide poor detection of negative or neutral comments.

The product closes the loop with an in-app flag for incorrect results. Review effort concentrates on low-confidence and disputed comments, where a human decision produces the most useful new evidence. Proposed changes to the lexicon, escalation policy, or model are tested against a fixed holdout set before release.

This turns model quality into a governed operating process: collect labels, analyse error patterns, adjust one policy, validate the change, and deploy only when the evidence improves.

The ingestion roadmap starts with the lowest-risk source and delays the most uncertain integration until the core loop is proven.

1. YouTube first: use the official YouTube Data API to validate the complete job, storage, analysis, and export path against a reliable source.
2. Instagram second: add the official Graph API once the pipeline works, including explicit handling for permissions, token expiry, and reauthorization.
3. Additional platforms behind a go/no-go gate: any source without a viable official comments API requires legal, terms-of-service, provenance, retention, and vendor-reliability review before production use.

The sequencing protects the MVP from spending its short timebox on the riskiest platform. It also makes non-official access a business decision with an owner, not an engineering shortcut hidden inside an adapter.

Authentication uses an external OIDC provider and signed JWTs. FastAPI extracts the user identity and enforces ownership in every job and result query. PostgreSQL row-level security provides defence in depth if an application query omits that filter.

Secrets are not stored beside jobs. The design places provider credentials in AWS Secrets Manager and grants workers the minimum permission required to retrieve the relevant secret. Large exports remain private and are exposed only through expiring URLs.

Operationally, queue depth is the leading indicator for burst pressure. Sustained backlog triggers horizontal worker scaling. CloudWatch captures structured logs, API latency, error rates, database pressure, job duration, and queue depth; alarms focus attention on conditions that threaten completion time or data integrity.

The production readiness checklist also includes questions the source architecture leaves for implementation: data retention and deletion, privacy assessment, duplicate and private-post handling, partial job completion, dead-letter policy, platform quotas, and cost per thousand comments.

The first architecture answers "what happened?" The larger product thesis asks how the same data foundation can support progressively better decisions.

The engagement produced a decision-ready architecture blueprint for a sentiment-analysis product under a demanding MVP constraint.

It established:

• an asynchronous job model suited to long-running, bursty work;
• a complete user journey from authenticated URL submission to downloadable results;
• explicit component boundaries across API, queue, workers, platform adapters, model, storage, and export;
• a phased source-integration plan that begins with official APIs;
• a cost-aware model strategy combining a local baseline with selective escalation;
• a human-in-the-loop quality and feedback process;
• defence-in-depth authorization, secret isolation, observability, retries, and scaling triggers;
• a four-stage maturity roadmap that lets the data foundation evolve without replacing the MVP.

This was a solution-design outcome, not a production-success report. No deployment metrics, realized ROI, live accuracy score, customer adoption figure, or cloud cost benchmark was available in the source document, so none is claimed.