Data Pipeline
Local-Sync Events
Mode & Sources
Enhancements & Build
This path flags pipeline_operator: venue categorization runs after extraction; inverted indexes /
embeddings / competitor finalize always afterward. Deep-research disk export and Phase 2.7 research_priority /
account_signals builds are omitted (use research jobs); ML inference parquet export stays off unless you POST
auto_ml_intelligence: true without pipeline_operator from the API.
ML Enrichment + Abstracts are forced on server-side
so Activities/Campaigns and abstract-gated venue tagging always participate.
When categorize finds zero venues queued for LLM batches (same junk/title rules as production),
processed_records is touched so the Pipeline table “Updated” clock reflects that verify pass.
Advanced: Individual ETL Operations
Source connectivity
Probes run in this runtime (laptop or Cloud Run web service). With the default deploy, the web service and all ETL jobs share one VPC NAT static IP — allowlist it on Navigator, Redshift, and HubSpot. Venues = Salesforce only; Redshift = EF3 abstracts; Navigator = bids/RFP.
Memory telemetry (RSS)
Peak RSS vs configured limit per job, plus a per-milestone flame chart from the [RSS] logs each ETL job emits. Delta = memory the phase ending at that milestone added; phase = its wall-clock duration. Use this to find which ETL phase drives the spike before resizing or rewriting it.
Execution History
| Job | Execution | Status | Started | Duration | |
|---|---|---|---|---|---|
| Loading… | |||||
Hot-Reload Production
Pull fresh data from GCS into the Cloud Run serving container and reload all caches.
Data Refreshers & ETL Scripts
Targeted, on-demand ways to get one dataset fresh without a full master_etl.
Each runs its ETL, syncs only the changed parquet to GCS, and (where applicable) hot-reloads
production — then deploy code-only with .\deploy.ps1. Run from the repo root in PowerShell.
Current State
Deploy Pipeline
Runs deploy.ps1 locally: test → build data bundle → docker build → push → Cloud Run rollout.
Docker Desktop must be running.
Revisions
| Revision | Created | CPU | Memory | Instances |
|---|---|---|---|---|
| Loading… | ||||
GCS Operations
Run History
| Job | Type | Status | Started | Duration | Log Lines | |
|---|---|---|---|---|---|---|
| Loading… | ||||||