safesight/docs/superpowers/specs/2026-04-01-region-behavior-recognition-design.md

# Region Behavior Recognition Design

## 1. Goal

Add region behavior recognition to the existing plugin-based DAG pipeline without breaking the current detection-oriented data model.

The first release must support:

- Distinguishing event type in OSD and event output
- Associating each event with one or more tracked persons
- Reporting how long the event has lasted

Target event types:

- `intrusion`
- `climb`
- `fall`
- `fight`

## 2. Problem Framing

The current project architecture is plugin-based and detection-centric:

- `tracker` maintains stable `track_id`
- AI nodes write object results into `frame->det`
- `osd` and `alarm` mainly consume object-like outputs

This is not sufficient for behavior events because behavior events are not object detections. A behavior event needs:

- event identity
- event type
- lifecycle state
- associated track ids
- duration
- optional region context

Encoding behavior events as synthetic detection classes would create a false model and couple downstream nodes to plugin-specific conventions. The design therefore introduces a dedicated behavior event data structure.

## 3. Recommended Architecture

The design separates rule-driven region events from temporal semantic events.

### 3.1 New and Updated Nodes

- `tracker`
  - Existing node
  - Keeps responsibility limited to stable `track_id` assignment
- `region_event`
  - New plugin
  - Handles rule-driven region events such as `intrusion` and rule-based `climb`
- `action_recog`
  - New plugin
  - Handles temporal semantic events such as `fall` and `fight`
- `event_fusion`
  - New plugin
  - Merges behavior events from upstream nodes into one stable output stream
- `osd`
  - Updated to render behavior events
- `alarm`
  - Updated to evaluate behavior events directly

### 3.2 First-Release Pipeline Shape

To minimize graph framework changes in the first iteration, behavior nodes should be chained serially while sharing the same `Frame` object:

```text
input -> preprocess -> person_det -> tracker -> region_event -> action_recog -> event_fusion -> osd -> publish -> alarm
```

This keeps the first release compatible with the current linear graph style while still enabling future branch-and-merge topologies if needed.

## 4. Data Model

### 4.1 New Behavior Event Types

Add a dedicated behavior event result to the frame model.

```cpp
enum class BehaviorEventType {
    Intrusion,
    Climb,
    Fall,
    Fight
};

enum class BehaviorEventStatus {
    Pending,
    Active,
    Ended
};

struct BehaviorEventItem {
    int event_id = -1;
    BehaviorEventType type = BehaviorEventType::Intrusion;
    BehaviorEventStatus status = BehaviorEventStatus::Pending;

    float score = 0.0f;

    Rect bbox{};
    std::vector<int> track_ids;

    uint64_t start_pts = 0;
    uint64_t last_pts = 0;
    uint64_t duration_ms = 0;

    std::string source;
    std::string region_id;
};

struct BehaviorEventResult {
    std::vector<BehaviorEventItem> items;
};
```

### 4.2 Frame Extension

Add this field to `Frame`:

```cpp
std::shared_ptr<BehaviorEventResult> behavior_events;
```

### 4.3 Design Rationale

This preserves a clean separation:

- `det` describes detected objects
- `behavior_events` describes interpreted events

This avoids overloading `cls_id`, avoids hidden `user_meta` contracts, and provides one stable contract for `event_fusion`, `osd`, `alarm`, and future APIs.

## 5. Node Responsibilities

### 5.1 `tracker`

Input:

- person detections in `frame->det`

Output:

- updated `frame->det` with stable `track_id`

Non-goals:

- no event interpretation
- no region semantics

### 5.2 `region_event`

Input:

- `frame->det`
- track ids from `tracker`
- configured regions and event rules

Output:

- appends region-derived events to `frame->behavior_events`

First-release responsibilities:

- `intrusion`
- rule-based `climb`

### 5.3 `action_recog`

Input:

- tracked targets from `frame->det`
- temporal history per `track_id`
- optional cropped target imagery or derived temporal features

Output:

- appends temporal behavior events to `frame->behavior_events`

First-release responsibilities:

- temporal-rule `fall`
- temporal-rule `fight`

Internal requirement:

- maintain per-track time windows and state machines

### 5.4 `event_fusion`

Input:

- behavior events produced by upstream behavior nodes

Output:

- normalized `frame->behavior_events`

Responsibilities:

- assign and maintain stable `event_id`
- merge duplicates from different behavior sources
- advance event lifecycle state
- keep one downstream-facing event contract

### 5.5 `osd`

Input:

- `frame->behavior_events`

Output:

- rendered overlays showing:
  - event type
  - associated track id or ids
  - duration
  - event box or associated target box

### 5.6 `alarm`

Input:

- `frame->behavior_events`

Output:

- event-driven alert actions

Responsibilities:

- rule matching by behavior type
- cooldown and suppression
- duration thresholds
- optional region filtering

## 6. First-Release Event Strategy

The first release should prioritize correctness of architecture and event lifecycle over ambitious model accuracy.

### 6.1 `intrusion`

Implementation mode:

- rule-based

Decision basis:

- tracked target enters configured ROI
- target remains inside for at least `min_duration_ms`

Optional filters:

- minimum object size
- class whitelist
- direction gate
- confidence threshold

### 6.2 `climb`

Implementation mode:

- rule-based in first release

Decision basis:

- tracked target approaches configured climb boundary
- target top, center, or bottom crosses boundary over time
- target exhibits meaningful vertical motion near the boundary

Optional filters:

- boundary side
- crossing ratio
- dwell time near boundary

This is intentionally framed as a region-crossing event first, not a semantic action model.

### 6.3 `fall`

Implementation mode:

- temporal rule/state-machine in first release

Decision basis:

- rapid centroid drop
- significant aspect-ratio change
- low-position persistence after transition

Suggested internal states:

- `normal`
- `suspicious`
- `active`
- `ended`

This keeps the data and pipeline correct while allowing replacement by a temporal model later.

### 6.4 `fight`

Implementation mode:

- temporal rule/state-machine in first release

Decision basis:

- two or more tracks stay in close proximity
- short-term strong relative motion or repeated box perturbation
- persistence over a short window

Risk note:

This event has the highest false-positive risk in a rule-only implementation. First release should be positioned as an early-warning event, not a high-certainty semantic classifier.

## 7. Event Lifecycle Rules

All behavior events should use the same lifecycle semantics.

- `Pending`
  - early evidence exists but activation threshold is not yet met
- `Active`
  - activation threshold is met and the event should be visible to OSD and alarm
- `Ended`
  - event was active and has now stopped, but may still be emitted briefly for lifecycle completion or external consumers

Core timestamps:

- `start_pts`: when evidence first appeared
- `last_pts`: latest frame contributing to this event
- `duration_ms`: derived from `start_pts` and `last_pts`

## 8. Configuration Direction

### 8.1 `region_event` Example Shape

```json
{
  "id": "region_evt",
  "type": "region_event",
  "events": [
    {
      "type": "intrusion",
      "region_id": "zone_a",
      "roi": {"x": 0.1, "y": 0.1, "w": 0.5, "h": 0.5},
      "min_duration_ms": 1000
    },
    {
      "type": "climb",
      "region_id": "fence_1",
      "line": {"x1": 0.2, "y1": 0.4, "x2": 0.8, "y2": 0.4},
      "min_duration_ms": 600
    }
  ]
}
```

### 8.2 `action_recog` Example Shape

```json
{
  "id": "action_evt",
  "type": "action_recog",
  "events": [
    {
      "type": "fall",
      "window_ms": 1500,
      "activate_duration_ms": 500
    },
    {
      "type": "fight",
      "window_ms": 1200,
      "min_tracks": 2,
      "activate_duration_ms": 400
    }
  ]
}
```

These are directional examples, not final schema commitments. Final schema should be kept explicit and event-specific rather than forced into one generic threshold bag.

## 9. OSD and Alarm Expectations

### 9.1 OSD

OSD must be able to render:

- event label such as `fall`, `fight`, `climb`, `intrusion`
- primary associated track id
- elapsed duration in milliseconds or seconds
- event box if available, otherwise tracked target box

### 9.2 Alarm

Alarm rules should support behavior-event inputs directly. Example dimensions:

- `event_types`
- `region_ids`
- `min_score`
- `min_duration_ms`
- `cooldown_ms`
- `per_track_cooldown_ms`

The alarm plugin should not reconstruct behavior semantics from detection classes.

## 10. Non-Goals for First Release

- full temporal deep-learning action model integration
- skeleton-based action recognition
- multi-camera cross-view identity reasoning
- perfect fight detection accuracy
- replacing existing object detection pipeline

## 11. Migration and Compatibility

The design should preserve compatibility with the current object-detection pipeline:

- existing graphs without behavior nodes must keep working
- `osd` and `alarm` behavior-event support should be additive
- behavior-event support should not require all pipelines to enable it

## 12. Implementation Order

Recommended implementation order:

1. Add behavior event data structures to the frame model
2. Extend `osd` and `alarm` contracts to consume behavior events
3. Implement `region_event`
4. Implement `action_recog` with temporal rule/state-machine version
5. Implement `event_fusion`
6. Add sample graph configs for intrusion, climb, fall, and fight

## 13. Risks

- If behavior events are encoded as synthetic detections, downstream semantics will become unstable
- If `fight` is treated as a high-certainty classifier in the first release, user expectations will exceed achievable accuracy
- If event lifecycle is not centralized, `osd` and `alarm` will drift into incompatible interpretations
- If action nodes directly own alert policy, the graph will become harder to tune and reuse

## 14. Decision Summary

The project should not implement one generic “action recognition plugin” first.

It should implement:

- one dedicated behavior event data model
- one rule-driven region event plugin
- one temporal action recognition plugin
- one event fusion plugin

This is the shortest path that matches the current project architecture and still supports first-release visibility of event type, associated target, and event duration.