UFOSINT Explorer

Unified UFO Sightings Database — Methodology

This is not raw data. UFOSINT Explorer presents a processed scientific analysis of six major UFO/UAP databases — 618,316 sighting records deduplicated, cross-referenced, geocode-verified, quality-scored, LLM-enriched, movement-classified, and emotion-analyzed using four transformer models. Every step of the pipeline is documented below and can be independently replicated from the source data using the open-source ufo-dedup pipeline. The web application source code is at ufosint-explorer.

Pipeline Architecture

The 17-step pipeline transforms ~2.56 million raw records from 6 sources into a unified, analysis-ready database:

            Raw Data (5 CSVs + 1 JSON + Reddit CSV)
                |
                v
            rebuild_db.py  (17 steps, ~30 min)
                |
                +-- Steps 1-7:    Import 6 sources (618,316 sightings)
                +-- Step 8:       SQL data quality fixes (~30 corrections)
                +-- Step 9:       Geocode pass 1 (GeoNames offline gazetteer)
                +-- Step 10:      Audit: fix bad geocodes + replay LLM location fixes
                +-- Step 11:      Geocode pass 2 (picks up audit-improved locations)
                +-- Steps 12-14:  Enrich, dedup, sentiment analysis
                +-- Step 15:      Derived analysis (shapes, quality, duration, hoax)
                +-- Step 16:      Replay cached enrichments (emotions, LLM extractions)
                +-- Step 17:      Export
                |
                v
            ufo_public.db  (618,316 sightings, 100 columns, 628 MB)
                    

Source Databases

Total raw records across all sources: ~2.56 million. After removing known overlaps at import time: 618,316.

LLM-Powered Data Quality Pipeline

After initial import and geocoding, the pipeline runs an AI-powered data quality audit using Google Gemini Flash. Results are cached for deterministic replay on future rebuilds.

Impact: +48,503 new correct map pins, +38,347 records above the quality threshold, 574,269 new structured field values extracted from text.

Geocoding

Locations are geocoded offline using the GeoNames gazetteer (cities with population ≥15,000). Three matching strategies with decreasing specificity:

1. Exact: city + state + country → coordinates (highest confidence)
2. City + country: ignores state, picks largest city by population
3. City only: global lookup, prefers matches in the expected country when a US/CA state code is present (v0.14 fix — prevents wrong-continent matches)

Coverage: 418,077 of 618,316 sightings (67.6%) have map coordinates. Two geocoding passes run: before and after the LLM location audit.

Quality Score (0–100)

Every sighting receives a quality score based on the richness and completeness of its data. Higher scores indicate more detailed, verifiable reports.

Distribution: 160,728 sightings (26%) score ≥60. Average score: 42.3. The structured fields row is where LLM extraction has the biggest impact — each newly extracted field adds 3 points.

Shape Normalization

Raw shape strings from 6 sources are normalized to 28 canonical shapes using a three-tier matcher:

1. Exact match against canonical list (286,826 matches)
2. Substring/alias match against 70+ extended mappings — e.g., "ovoid" → Oval, "V-shape" → Chevron, "torpedo" → Cigar (52,448 matches)
3. Fuzzy match via rapidfuzz at ≥85% similarity (41 matches)

28 canonical shapes: Sphere, Disc, Triangle, Cigar, Oval, Circle, Light, Fireball, Cylinder, Diamond, Rectangle, Chevron, Cross, Teardrop, Star, Egg, Cone, Cube, Saucer, Boomerang, Flash, Formation, Changing, Crescent, Cloud, Dome, Unknown, Other.

Coverage: 343,602 sightings (55.6%) have a standardized shape.

Duration Parsing

Raw duration strings are parsed to seconds using a multi-strategy parser:

• Natural language: "5 minutes" → 300s, "about 2 hours" → 7200s
• UFOCAT codes: "B" (brief) → 3s, "M" (medium) → 120s, "H" (hour) → 3600s
• Bare numbers: "5" → 300s (UFOCAT convention: decimal minutes)
• Ranges: "5-10 minutes" → 450s (midpoint)

Parsed durations are bucketed: instant (<5s), seconds (<60s), minutes (<1h), hours (<24h), days (≥24h).

Coverage: 232,438 sightings (37.6%) have parsed durations.

Emotion & Sentiment Analysis

Four models classify the emotional content of sighting narratives:

Nuclear Proximity Analysis

Haversine distance computed from every geocoded sighting to the nearest of 50 nuclear-relevant facilities (military bases, national labs, nuclear test sites, weapons storage areas).

Results: 35,203 sightings within 50 km of a nuclear site. 69,912 within 100 km.

Supplemented by 14 crash-retrieval cases and 35 nuclear encounter records from the UAP Gerb research bundle.

Movement & Behavior Classification

Regex-based extraction from narrative text identifies 10 movement categories and 14 behavior tags:

• Movement: hovering, linear, erratic, accelerating, rotating, ascending, descending, vanished, followed, landed
• Behavior: hovering, silent, bright, pulsing, rotating, zigzag, vanished, accelerated, split, merged, formation, chased, followed, landed

Coverage: 250,820 sightings (40.6%) have at least one movement category.

Cross-Source Deduplication

A three-tier engine flags potential duplicates across sources. No records are deleted — duplicates are flagged for review.

Total: 126,729 duplicate candidate pairs across 618,316 sightings.

Content Policy & Privacy

The public database (ufo_public.db) contains only derived, non-copyrighted fields. Raw narrative text from NUFORC, MUFON, UFOCAT, UPDB, and UFO-search is stripped during export. Reddit sighting descriptions are LLM-generated summaries (transformative derivative works), not original Reddit posts.

No personally identifiable information (PII) is published. The witness_names column is NULL in the public export.

How to Reproduce This Database

            pip install -r requirements-etl.txt
            python geocode.py --download              # one-time gazetteer (~30 MB)
            python rebuild_db.py                       # 17-step pipeline (~30 min)
            python emotions.py                         # GPU emotion classification (~35 min)
            python run_enrich.py --limit 378000        # LLM field extraction (~$8, optional)
            python run_enrich.py --apply               # apply extractions to DB
            python export_public.py                    # clean public export
                

LLM-derived results (location normalization, field extraction, emotion classification) are cached to CSV files in data/output/ and replayed automatically on future rebuilds without re-calling APIs or re-running GPU inference.