safesight/docs/superpowers/specs/2026-04-01-region-behavior-recognition-design.md

Region Behavior Recognition Design

1. Goal

Add region behavior recognition to the existing plugin-based DAG pipeline without breaking the current detection-oriented data model.

The first release must support:

• Distinguishing event type in OSD and event output
• Associating each event with one or more tracked persons
• Reporting how long the event has lasted

Target event types:

• intrusion
• climb
• fall
• fight

2. Problem Framing

The current project architecture is plugin-based and detection-centric:

• tracker maintains stable track_id
• AI nodes write object results into frame->det
• osd and alarm mainly consume object-like outputs

This is not sufficient for behavior events because behavior events are not object detections. A behavior event needs:

• event identity
• event type
• lifecycle state
• associated track ids
• duration
• optional region context

Encoding behavior events as synthetic detection classes would create a false model and couple downstream nodes to plugin-specific conventions. The design therefore introduces a dedicated behavior event data structure.

3. Recommended Architecture

The design separates rule-driven region events from temporal semantic events.

3.1 New and Updated Nodes

• tracker
  • Existing node
  • Keeps responsibility limited to stable track_id assignment
• region_event
  • New plugin
  • Handles rule-driven region events such as intrusion and rule-based climb
• action_recog
  • New plugin
  • Handles temporal semantic events such as fall and fight
• event_fusion
  • New plugin
  • Merges behavior events from upstream nodes into one stable output stream
• osd
  • Updated to render behavior events
• alarm
  • Updated to evaluate behavior events directly

3.2 First-Release Pipeline Shape

To minimize graph framework changes in the first iteration, behavior nodes should be chained serially while sharing the same Frame object:

input -> preprocess -> person_det -> tracker -> region_event -> action_recog -> event_fusion -> osd -> publish -> alarm

This keeps the first release compatible with the current linear graph style while still enabling future branch-and-merge topologies if needed.

4. Data Model

4.1 New Behavior Event Types

Add a dedicated behavior event result to the frame model.

enum class BehaviorEventType {
    Intrusion,
    Climb,
    Fall,
    Fight
};

enum class BehaviorEventStatus {
    Pending,
    Active,
    Ended
};

struct BehaviorEventItem {
    int event_id = -1;
    BehaviorEventType type = BehaviorEventType::Intrusion;
    BehaviorEventStatus status = BehaviorEventStatus::Pending;

    float score = 0.0f;

    Rect bbox{};
    std::vector<int> track_ids;

    uint64_t start_pts = 0;
    uint64_t last_pts = 0;
    uint64_t duration_ms = 0;

    std::string source;
    std::string region_id;
};

struct BehaviorEventResult {
    std::vector<BehaviorEventItem> items;
};

4.2 Frame Extension

Add this field to Frame:

std::shared_ptr<BehaviorEventResult> behavior_events;

4.3 Design Rationale

This preserves a clean separation:

• det describes detected objects
• behavior_events describes interpreted events

This avoids overloading cls_id, avoids hidden user_meta contracts, and provides one stable contract for event_fusion, osd, alarm, and future APIs.

5. Node Responsibilities

5.1 tracker

Input:

• person detections in frame->det

Output:

• updated frame->det with stable track_id

Non-goals:

• no event interpretation
• no region semantics

5.2 region_event

Input:

• frame->det
• track ids from tracker
• configured regions and event rules

Output:

• appends region-derived events to frame->behavior_events

First-release responsibilities:

• intrusion
• rule-based climb

5.3 action_recog

Input:

• tracked targets from frame->det
• temporal history per track_id
• optional cropped target imagery or derived temporal features

Output:

• appends temporal behavior events to frame->behavior_events

First-release responsibilities:

• temporal-rule fall
• temporal-rule fight

Internal requirement:

• maintain per-track time windows and state machines

5.4 event_fusion

Input:

• behavior events produced by upstream behavior nodes

Output:

• normalized frame->behavior_events

Responsibilities:

• assign and maintain stable event_id
• merge duplicates from different behavior sources
• advance event lifecycle state
• keep one downstream-facing event contract

5.5 osd

Input:

• frame->behavior_events

Output:

• rendered overlays showing:
  • event type
  • associated track id or ids
  • duration
  • event box or associated target box

5.6 alarm

Input:

• frame->behavior_events

Output:

• event-driven alert actions

Responsibilities:

• rule matching by behavior type
• cooldown and suppression
• duration thresholds
• optional region filtering

6. First-Release Event Strategy

The first release should prioritize correctness of architecture and event lifecycle over ambitious model accuracy.

6.1 intrusion

Implementation mode:

• rule-based

Decision basis:

• tracked target enters configured ROI
• target remains inside for at least min_duration_ms

Optional filters:

• minimum object size
• class whitelist
• direction gate
• confidence threshold

6.2 climb

Implementation mode:

• rule-based in first release

Decision basis:

• tracked target approaches configured climb boundary
• target top, center, or bottom crosses boundary over time
• target exhibits meaningful vertical motion near the boundary

Optional filters:

• boundary side
• crossing ratio
• dwell time near boundary

This is intentionally framed as a region-crossing event first, not a semantic action model.

6.3 fall

Implementation mode:

• temporal rule/state-machine in first release

Decision basis:

• rapid centroid drop
• significant aspect-ratio change
• low-position persistence after transition

Suggested internal states:

• normal
• suspicious
• active
• ended

This keeps the data and pipeline correct while allowing replacement by a temporal model later.

6.4 fight

Implementation mode:

• temporal rule/state-machine in first release

Decision basis:

• two or more tracks stay in close proximity
• short-term strong relative motion or repeated box perturbation
• persistence over a short window

Risk note:

This event has the highest false-positive risk in a rule-only implementation. First release should be positioned as an early-warning event, not a high-certainty semantic classifier.

7. Event Lifecycle Rules

All behavior events should use the same lifecycle semantics.

• Pending
  • early evidence exists but activation threshold is not yet met
• Active
  • activation threshold is met and the event should be visible to OSD and alarm
• Ended
  • event was active and has now stopped, but may still be emitted briefly for lifecycle completion or external consumers

Core timestamps:

• start_pts: when evidence first appeared
• last_pts: latest frame contributing to this event
• duration_ms: derived from start_pts and last_pts

8. Configuration Direction

8.1 region_event Example Shape

{
  "id": "region_evt",
  "type": "region_event",
  "events": [
    {
      "type": "intrusion",
      "region_id": "zone_a",
      "roi": {"x": 0.1, "y": 0.1, "w": 0.5, "h": 0.5},
      "min_duration_ms": 1000
    },
    {
      "type": "climb",
      "region_id": "fence_1",
      "line": {"x1": 0.2, "y1": 0.4, "x2": 0.8, "y2": 0.4},
      "min_duration_ms": 600
    }
  ]
}

8.2 action_recog Example Shape

{
  "id": "action_evt",
  "type": "action_recog",
  "events": [
    {
      "type": "fall",
      "window_ms": 1500,
      "activate_duration_ms": 500
    },
    {
      "type": "fight",
      "window_ms": 1200,
      "min_tracks": 2,
      "activate_duration_ms": 400
    }
  ]
}

These are directional examples, not final schema commitments. Final schema should be kept explicit and event-specific rather than forced into one generic threshold bag.

9. OSD and Alarm Expectations

9.1 OSD

OSD must be able to render:

• event label such as fall, fight, climb, intrusion
• primary associated track id
• elapsed duration in milliseconds or seconds
• event box if available, otherwise tracked target box

9.2 Alarm

Alarm rules should support behavior-event inputs directly. Example dimensions:

• event_types
• region_ids
• min_score
• min_duration_ms
• cooldown_ms
• per_track_cooldown_ms

The alarm plugin should not reconstruct behavior semantics from detection classes.

10. Non-Goals for First Release

• full temporal deep-learning action model integration
• skeleton-based action recognition
• multi-camera cross-view identity reasoning
• perfect fight detection accuracy
• replacing existing object detection pipeline

11. Migration and Compatibility

The design should preserve compatibility with the current object-detection pipeline:

• existing graphs without behavior nodes must keep working
• osd and alarm behavior-event support should be additive
• behavior-event support should not require all pipelines to enable it

12. Implementation Order

Recommended implementation order:

1. Add behavior event data structures to the frame model
2. Extend osd and alarm contracts to consume behavior events
3. Implement region_event
4. Implement action_recog with temporal rule/state-machine version
5. Implement event_fusion
6. Add sample graph configs for intrusion, climb, fall, and fight

13. Risks

• If behavior events are encoded as synthetic detections, downstream semantics will become unstable
• If fight is treated as a high-certainty classifier in the first release, user expectations will exceed achievable accuracy
• If event lifecycle is not centralized, osd and alarm will drift into incompatible interpretations
• If action nodes directly own alert policy, the graph will become harder to tune and reuse

14. Decision Summary

The project should not implement one generic “action recognition plugin” first.

It should implement:

• one dedicated behavior event data model
• one rule-driven region event plugin
• one temporal action recognition plugin
• one event fusion plugin

This is the shortest path that matches the current project architecture and still supports first-release visibility of event type, associated target, and event duration.