GAF Feature Pipeline — ML Case Study

01 The Hypothesis

Can a convolutional neural network learn to predict short-term price direction by recognizing visual patterns in a GAF image?

A GAF matrix is a compact fingerprint of a price series' internal temporal structure — two windows with similar momentum, reversal, and consolidation dynamics produce visually similar matrices. If patterns that look like today's consistently preceded a specific outcome, a CNN should learn the association.

Target: given the current 120-bar GAF, output a statement like "80% probability this market moves up 2–5% over the next 5 bars."

fig 0 — the idea

  price series (120 bars)
           │
           ▼
     GAF transform
           │
           ▼
  120×120 matrix "image"
           │
           ▼
     CNN classifier
           │
           ▼
  "80% probability this market
   moves +2–5% over next 5 bars"

02 What Is a GAF?

A Gramian Angular Field converts a 1-D time series into a 2-D matrix that encodes temporal correlation: each cell [i, j] expresses the relationship between the price at time i and time j. The result is image-shaped — exactly what CNNs are built to consume.

The variant used is the Gramian Angular Summation Field: result[i,j] = cos(phi_i + phi_j). Four steps, implemented in GAF.cs:

Min-max scale the series to [-1, 1]
Clamp floating-point overshoot so acos stays valid
Arc-cosine each value: phi[i] = acos(scaled[i])
Build the N×N matrix: cos(phi[i] + phi[j])

fig A1 — GAF algorithm (GAF.cs)

  float[] series (N values)
         │
         ▼
  ┌─────────────────────────────────────┐
  │  Min-Max scale to [-1, 1]           │
  │                                     │
  │  min  = series.Min()                │
  │  max  = series.Max()                │
  │  scaled[i] = (2*x[i] - max - min)   │
  │              / (max - min)          │
  └──────────────────┬──────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────┐
  │  Clamp to [-1, 1]                   │
  │                                     │
  │  if scaled[i] >  1.0 → set  1.0    │
  │  if scaled[i] < -1.0 → set -1.0    │
  └──────────────────┬──────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────┐
  │  Arc-cosine transform               │
  │                                     │
  │  phi[i] = acos(scaled[i])           │
  │           result in [0, π]          │
  └──────────────────┬──────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────┐
  │  Build N×N output matrix            │
  │                                     │
  │  for i in 0..N-1:                   │
  │    for j in 0..N-1:                 │
  │      result[i,j] = cos(phi[i]       │
  │                      + phi[j])      │
  └──────────────────┬──────────────────┘
                     │
                     ▼
             float[N, N]  (Result)

03 Windows & Labels

Each training sample is a window of 125 consecutive price bars, split 120 + 5: the first 120 bars become four GAF matrices (Open, High, Low, Close — four "channels", like RGB in an image), and the last 5 bars become the label.

The label asks: what did price do next? The percent difference between the last analysis close and the average of the 5 future closes is bucketed into one of 13 classes, from DownGreaterThan20Percent through Neutral to UpGreaterThan20Percent, then one-hot encoded as float[13]. Moves beyond ±30% are treated as data errors and filtered.

fig A2 — window split & label derivation (GafFeature.cs)

  125 PriceData bars
  ┌──────────────────────────────────┬───────────┐
  │  Analysis period  (bars 0–119)   │  Feature  │
  │         120 bars                 │  period   │
  │                                  │  5 bars   │
  │  GAF(Open)   → float[120,120]    │ (120–124) │
  │  GAF(High)   → float[120,120]    │           │
  │  GAF(Low)    → float[120,120]    │           │
  │  GAF(Close)  → float[120,120]    │           │
  └──────────────────────────────────┴─────┬─────┘
                                           │
                          ┌────────────────┘
                          ▼
            lastClose = bar[119].Close
            avgFuture = avg(bars[120..124].Close)
            pctDiff   = (avgFuture - lastClose) / lastClose
                          │
                          ▼
            ┌──────────────────────────────────┐
            │  Bucket pctDiff into 13 classes  │
            │                                  │
            │  > +30%   → (no label, filtered) │
            │  +20–30%  → UpGreaterThan20      │
            │  +10–20%  → Up10To20             │
            │   +5–10%  → Up5To10              │
            │   +2–5%   → Up2To5               │
            │   +1–2%   → Up1To2               │
            │  +0.5–1%  → Up05To1              │
            │ ±0.5%     → Neutral              │
            │  -0.5–1%  → Down05To1            │
            │   -1–2%   → Down1To2             │
            │   -2–5%   → Down2To5             │
            │   -5–10%  → Down5To10            │
            │  -10–20%  → Down10To20           │
            │  -20–30%  → DownGreaterThan20    │
            │  < -30%   → (no label, filtered) │
            └─────────────┬────────────────────┘
                          │
                          ▼
            ChartFeatureInformation (one bool set)
                          │
                          ▼
            OneHot() → float[13]
            e.g. Neutral → [0,0,0,0,0,0,1,0,0,0,0,0,0]

04 Bulk Generation

A C# ETL task (CreateGafChartFeaturesInDb) drives generation across every market in MongoDB. For each ticker and timeframe it slides the 125-bar window one bar at a time across all available history — every position becomes one labeled training sample, bulk-inserted as a MongoDB document holding all four matrices and the one-hot label.

Markets with fewer than 500 bars are skipped — not enough signal
Daily, weekly, and monthly passes per market
Optional parallel execution (Parallel.ForEach, max degree 10)

fig A3 — sliding window (CreateGafChartFeaturesInDb.cs)

  All price bars for one ticker + TimeLevel
  ┌───┬───┬───┬───┬───┬───┬─ ─ ─ ┬───┐
  │ 0 │ 1 │ 2 │ 3 │...│124│      │ N │
  └───┴───┴───┴───┴───┴───┴─ ─ ─ ┴───┘

  i=0   [0 ──────────────── 124]
  i=1       [1 ──────────────── 125]
  i=2           [2 ──────────────── 126]
  ...
  i=N-125               [N-125 ─────── N-1]

  Each window → GafFeature → GafChartFeatureResult
                                    │
  After all windows complete:       │
                                    ▼
                       List<GafChartFeature>
                                    │
                                    ▼
               MongoPriceDataFeatureDataStorage
                      .Insert(ticker, timeLevel, list)
                                    │
                                    ▼
                    MongoDB: GafChartFeature collection

05 CNN Training

The Python side read the matrices from MongoDB and rendered them to PNG images — pixel intensity mapped from each cell's value in [-1, +1]. Images were written into a folder tree organized by outcome class, split 70% train / 30% test chronologically, so the test set represents genuinely unseen future data.

A standard Keras/TensorFlow CNN classifier trained on the image tree, with the 13 class folders providing labels automatically. The intended production loop — current window → GAF → image → CNN → trade signal above an 80% confidence threshold — was designed but never built.

fig A4 — full pipeline (C# ETL → Python CNN)

  ┌─────────────────────────────────────────┐
  │  C# — SharpTechnicals.FileGenerator     │
  └─────────────────────────────────────────┘

  Program.cs
       │
       ▼
  CreateGafChartFeaturesInDb.Run()
       │
       ├─ MongoGenericDataStorage<MarketData>
       ├─ MongoPriceDataStorage
       ├─ TiingoConnector
       └─ MarketUniverse
              │
              ▼
       GetUpdateableMarkets()
              │
              ▼
       Filter: Ticker == "SPY"
              │
              ▼
       ProcessMarket.Process(mrkt)
              │
              ├──► Daily bars
              ├──► Weekly bars
              └──► Monthly bars
                       │
                       ▼
              bars.Count < 500?  ── YES ──► (skip)
                       │ NO
                       ▼
              Slide 125-bar window (step = 1)
                       │
                       ▼
              GafFeature(window)
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
     GAF on Open/High/    ChartFeatureInfo
     Low/Close (120 bars) from next 5 bars
     → 4× float[120,120]  → OneHot float[13]
             │                   │
             └─────────┬─────────┘
                       ▼
            GafChartFeature → MongoDB

  ┌─────────────────────────────────────────┐
  │  Python — external                      │
  └─────────────────────────────────────────┘

  Read GafChartFeature from MongoDB
       │
       ▼
  Render float[120,120] → PNG images
       │
       ▼
  Split chronologically: 70% TRAIN / 30% TEST
       │
       ▼
  Folder tree by outcome class
       ├── TRAIN/Up2To5Percent/...
       └── TEST/Down1To2Percent/...
                  │
                  ▼
  CNN training (Keras / TensorFlow)
  softmax over 13 outcome classes
                  │
                  ▼
  ┌──────────────────────────────────────┐
  │  Intended inference (never built)    │
  │  window → GAF → PNG → CNN predict    │
  │  if P(class) > 0.80 → trade signal   │
  └──────────────────────────────────────┘
                  │
                  ▼
       Results were lackluster.
       Experiment discontinued.

06 Outcome — Why It Was Shut Down

The full pipeline ran end-to-end: dataset generation, image export, CNN training. The model did not generalize well enough to be useful as a signal, and the experiment was discontinued rather than tuned into the ground. Likely contributing factors:

Training data was initially limited to SPY only — limited pattern variety
13-class label granularity was probably too fine for the data volume
GAF matrices encode relative temporal structure but lose absolute price level
A 5-bar forward average is a noisy label — one large bar can flip the class

The engineering takeaway: the pipeline architecture — C# feature generation, MongoDB persistence, Python training — worked exactly as designed. The hypothesis didn't survive contact with the data, and knowing when to stop is part of the job.

> next: cat /projects/call-center — the GenAI POC