Explainable Reputation Estimation from Web Service Reviews ^†

Abstract

Star ratings alone are noisy, manipulable, and ignore aspect-level sentiment. We present Scrape2Repute, a compact and reproducible pipeline that ingests Yelp reviews under policy constraints; cleans and normalises text/metadata; learns a calibrated text sentiment per review; fuses stars and text via a tunable hybrid label; downweights suspicious reviews with unsupervised anomaly scoring; and aggregates evidence into a time-decayed business reputation with uncertainty bounds. The system is explainable (top-k rationales, aspect summaries), runs on commodity hardware, and ships with CLI/GUI. On the Yelp Open Dataset, we show strong predictive validity for forecasting future ratings and stable behaviour under sensitivity sweeps. We release implementation and an ethics checklist for compliant use.

1. Introduction

User reviews on Yelp influence consumer decisions, yet naive star averaging ignores textual nuance, temporal drift, and manipulation risk. An auditable solution should (1) integrate textual sentiment with stars; (2) reduce the impact of stale or suspicious evidence; (3) expose explanations; and (4) be reproducible and policy-compliant.

Contributions: Scrape2Repute provides (i) an end-to-end pipeline with minimal dependencies; (ii) a hybrid per-review label combining normalised stars with a calibrated text score; (iii) a lightweight anomaly screen; (iv) time-decayed aggregation with confidence intervals; and (v) explainability via n-gram/token attributions and aspect summaries. We release CLI/GUI entry points and an ethics and compliance checklist tailored to data. An overview of the end-to-end architecture is shown in Figure 1.

Paper organisation. Section 3 details the pipeline; Section 4 reports predictive validity and sensitivity; Section 5 states compliance; and Section 6 concludes.

2. Related Work

Work on online reputation blends ratings, text, and interactions. Collaborative filtering and trust modelling improve robustness and mitigate bias across contexts [1,2]. Reputation dynamics, such as inertia, motivate temporal and adaptive treatment [3]. For web services, QoS-aware models and matrix factorisation fuse performance indicators with reputation signals [4,5]; integrity is improved by graph-based filtering of dishonest raters and credibility modelling [6,7]. Personalised reputation adapts scores to user history [8]. Our pipeline operationalises these ideas end-to-end on Yelp: calibrated text+stars fusion, unsupervised anomaly dampening, and time-decayed aggregation with uncertainty.

3. Methodology

Our pipeline comprises five stages: Acquisition (Yelp-compliant loading), Preprocessing (normalisation, dedup), Modelling (text classifier + calibration; hybrid fusion), Screening (unsupervised anomalies), and Aggregation (time decay with uncertainty).

3.1. Data Access and Normalisation

We use the Open Dataset [9], focusing on review and business. For each review we keep (review_id, business_id, user_id, stars, date, text). Stars are normalised as $s_{\star }=(\mathrm{stars}-1)/4\in [0,1]$, with timestamps $t_i$. We remove empty/near-empty texts, strip HTML, lowercase, normalise whitespace/emoji, optionally filter to English, and deduplicate by hashing normalised text.

3.2. Text Modelling and Calibration

Let $x_i$ denote a vectorised review (TF–IDF). A logistic classifier produces a calibrated probability $s_{\mathrm{text}}\left(x_i\right)=\sigma (w^{\top }{x}_i+b)\in [0,1]$ (Platt/isotonic). Sentence embeddings with a non-linear classifier can replace TF–IDF where resources permit.

3.3. Hybrid, Anomaly, and Aggregation

$$\begin{array}{cc}\hfill y_i\left(\alpha \right)& =\alpha s_{\star ,i}+(1-\alpha )s_{\mathrm{text},i},\alpha \in [0,1],\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill w_i\left(t\right)& =exp\left(-\lambda {[t-t_i]}_{+}\right){\gamma }_i,{\gamma }_i=(1-a_i),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \widehat{R}\left(t\right)& ={\displaystyle \frac{{\sum }_i{w}_i\left(t\right)y_i}{{\sum }_i{w}_i\left(t\right)}},\widehat{R}\pm 1.96\sqrt{\widehat{R}(1-\widehat{R})/{\sum }_i{w}_i}.\hfill \end{array}$$

(3)

An Isolation Forest estimates soft anomaly $a_i\in [0,1]$ from length, repetition, burstiness, and (where legal) account cues [10]. The robustness weight ${\gamma }_i$ attenuates suspicious reviews. We select $\alpha$ via cross-validation or stability criteria.

4. Experiments and Results

4.1. Setup

We train a TF-IDF + Logistic classifier with weak supervision from extreme stars (normalised $\ge 0.7$ positive, $\le 0.3$ negative) and Platt calibration (3-fold; 50% calibration split). Inference runs in chunks on the cleaned review file, followed by Bayesian reputation aggregation and anomaly estimation.

Data Scale and Footprint: We processed $\approx 6.99$ M reviews; the 90-day split retained 7059 businesses with $\min \_\mathrm{early}\ge 5$ and $\min \_\mathrm{future}\ge 5$. Training (TF-IDF + LR + Platt) took ∼88 min; the sensitivity sweep ∼30 min; anomaly scoring ∼15 min; and other stages $<2$ min each on a commodity workstation. These figures suggest that the pipeline is practical for routine, offline recalibration at scale.

4.2. Text Model Quality

On a held-out sample (n = 7846), accuracy is 0.875; per-class metrics are in

Table 1. Calibrated text classifier (TF-IDF + LR).

Class	Precision	Recall	F1
$-1$ (neg)	0.832	0.885	0.857
0 (neu)	0.543	0.256	0.348
$+1$ (pos)	0.913	0.965	0.938
Accuracy	0.875 (macro F1 0.715)

. The positive class shows strong precision/recall; neutral remains challenging under weak supervision.

4.3. Predictive Validity (Early → Future)

We test whether early hybrid scores forecast future stars at the business level (90-day horizon; $\min \_\mathrm{early}\ge 5$, $\min \_\mathrm{future}\ge 5$; 7059 businesses; see Figure 2 and Figure 3a). Weighted mean future star01 = 0.7239; Pearson r = 0.7553 (weighted $0.7652$); Spearman $\rho$ = 0.6737 (weighted $0.6873$); and top-decile mean = 0.9069 ($+25.27\%$ vs. overall).

4.4. Distribution and Sensitivity

Figure 3 summarises predictive validity and the distribution of Bayesian reputation (mode ∼70); see panels (a, b).

We sweep half-life $\in \{180,365,540,720\}$, suspicious-weight $\in \{0.1,0.2,0.3,0.5,0.8,1.0\}$, and $\alpha \in \{10,20,30,50,80\}$;

Table 2. Sensitivity summary (90-day horizon).

Config	HL	SW	$\mathit{\alpha }$	Metric
Baseline	365	0.3	20	$r=0.7457$, lift $=21.66\%$
Best-r	540	1.0	10	$r=0.7664$, lift $=24.41\%$
Best-lift	720	1.0	10	$r=0.7663$, lift $=24.49\%$

contrasts the baseline (HL = 365, SW = 0.3, $\alpha =20$) with best settings and—practically—recommends HL 540–720, SW $0.5$–$1.0$, and $\alpha \in \{10,20\}$.

Baseline contrast: A stars-only aggregator is included as a baseline; the hybrid shows better decile calibration (Figure 2) and higher predictive validity across the sweep (Table 2).

Reproducibility: Code, configs, fixed seeds, and artefacts enable end-to-end replication.

Practical impact: The pipeline supports periodic offline recalibration and lightweight deployment for marketplace ranking, vendor monitoring, and early-warning alerts.

5. Ethics, Compliance

We use the Yelp Open Dataset under its research terms [9]; we avoid PII, apply rate limiting, and honour robots and policies.

6. Conclusions

Scrape2Repute offers an auditable pathway from Yelp reviews to a time-aware, anomaly-robust reputation with explanations. Predictive validity is strong (Pearson $\approx 0.76$ at 90-day horizon) and stable across decay/anomaly settings. Future work includes multilingual aspect extraction, category-specific calibration, and user studies on explanation usefulness.