diff --git a/docs/superpowers/specs/2026-04-01-region-behavior-recognition-design.md b/docs/superpowers/specs/2026-04-01-region-behavior-recognition-design.md new file mode 100644 index 0000000..133d788 --- /dev/null +++ b/docs/superpowers/specs/2026-04-01-region-behavior-recognition-design.md @@ -0,0 +1,449 @@ +# Region Behavior Recognition Design + +## 1. Goal + +Add region behavior recognition to the existing plugin-based DAG pipeline without breaking the current detection-oriented data model. + +The first release must support: + +- Distinguishing event type in OSD and event output +- Associating each event with one or more tracked persons +- Reporting how long the event has lasted + +Target event types: + +- `intrusion` +- `climb` +- `fall` +- `fight` + +## 2. Problem Framing + +The current project architecture is plugin-based and detection-centric: + +- `tracker` maintains stable `track_id` +- AI nodes write object results into `frame->det` +- `osd` and `alarm` mainly consume object-like outputs + +This is not sufficient for behavior events because behavior events are not object detections. A behavior event needs: + +- event identity +- event type +- lifecycle state +- associated track ids +- duration +- optional region context + +Encoding behavior events as synthetic detection classes would create a false model and couple downstream nodes to plugin-specific conventions. The design therefore introduces a dedicated behavior event data structure. + +## 3. Recommended Architecture + +The design separates rule-driven region events from temporal semantic events. + +### 3.1 New and Updated Nodes + +- `tracker` + - Existing node + - Keeps responsibility limited to stable `track_id` assignment +- `region_event` + - New plugin + - Handles rule-driven region events such as `intrusion` and rule-based `climb` +- `action_recog` + - New plugin + - Handles temporal semantic events such as `fall` and `fight` +- `event_fusion` + - New plugin + - Merges behavior events from upstream nodes into one stable output stream +- `osd` + - Updated to render behavior events +- `alarm` + - Updated to evaluate behavior events directly + +### 3.2 First-Release Pipeline Shape + +To minimize graph framework changes in the first iteration, behavior nodes should be chained serially while sharing the same `Frame` object: + +```text +input -> preprocess -> person_det -> tracker -> region_event -> action_recog -> event_fusion -> osd -> publish -> alarm +``` + +This keeps the first release compatible with the current linear graph style while still enabling future branch-and-merge topologies if needed. + +## 4. Data Model + +### 4.1 New Behavior Event Types + +Add a dedicated behavior event result to the frame model. + +```cpp +enum class BehaviorEventType { + Intrusion, + Climb, + Fall, + Fight +}; + +enum class BehaviorEventStatus { + Pending, + Active, + Ended +}; + +struct BehaviorEventItem { + int event_id = -1; + BehaviorEventType type = BehaviorEventType::Intrusion; + BehaviorEventStatus status = BehaviorEventStatus::Pending; + + float score = 0.0f; + + Rect bbox{}; + std::vector track_ids; + + uint64_t start_pts = 0; + uint64_t last_pts = 0; + uint64_t duration_ms = 0; + + std::string source; + std::string region_id; +}; + +struct BehaviorEventResult { + std::vector items; +}; +``` + +### 4.2 Frame Extension + +Add this field to `Frame`: + +```cpp +std::shared_ptr behavior_events; +``` + +### 4.3 Design Rationale + +This preserves a clean separation: + +- `det` describes detected objects +- `behavior_events` describes interpreted events + +This avoids overloading `cls_id`, avoids hidden `user_meta` contracts, and provides one stable contract for `event_fusion`, `osd`, `alarm`, and future APIs. + +## 5. Node Responsibilities + +### 5.1 `tracker` + +Input: + +- person detections in `frame->det` + +Output: + +- updated `frame->det` with stable `track_id` + +Non-goals: + +- no event interpretation +- no region semantics + +### 5.2 `region_event` + +Input: + +- `frame->det` +- track ids from `tracker` +- configured regions and event rules + +Output: + +- appends region-derived events to `frame->behavior_events` + +First-release responsibilities: + +- `intrusion` +- rule-based `climb` + +### 5.3 `action_recog` + +Input: + +- tracked targets from `frame->det` +- temporal history per `track_id` +- optional cropped target imagery or derived temporal features + +Output: + +- appends temporal behavior events to `frame->behavior_events` + +First-release responsibilities: + +- temporal-rule `fall` +- temporal-rule `fight` + +Internal requirement: + +- maintain per-track time windows and state machines + +### 5.4 `event_fusion` + +Input: + +- behavior events produced by upstream behavior nodes + +Output: + +- normalized `frame->behavior_events` + +Responsibilities: + +- assign and maintain stable `event_id` +- merge duplicates from different behavior sources +- advance event lifecycle state +- keep one downstream-facing event contract + +### 5.5 `osd` + +Input: + +- `frame->behavior_events` + +Output: + +- rendered overlays showing: + - event type + - associated track id or ids + - duration + - event box or associated target box + +### 5.6 `alarm` + +Input: + +- `frame->behavior_events` + +Output: + +- event-driven alert actions + +Responsibilities: + +- rule matching by behavior type +- cooldown and suppression +- duration thresholds +- optional region filtering + +## 6. First-Release Event Strategy + +The first release should prioritize correctness of architecture and event lifecycle over ambitious model accuracy. + +### 6.1 `intrusion` + +Implementation mode: + +- rule-based + +Decision basis: + +- tracked target enters configured ROI +- target remains inside for at least `min_duration_ms` + +Optional filters: + +- minimum object size +- class whitelist +- direction gate +- confidence threshold + +### 6.2 `climb` + +Implementation mode: + +- rule-based in first release + +Decision basis: + +- tracked target approaches configured climb boundary +- target top, center, or bottom crosses boundary over time +- target exhibits meaningful vertical motion near the boundary + +Optional filters: + +- boundary side +- crossing ratio +- dwell time near boundary + +This is intentionally framed as a region-crossing event first, not a semantic action model. + +### 6.3 `fall` + +Implementation mode: + +- temporal rule/state-machine in first release + +Decision basis: + +- rapid centroid drop +- significant aspect-ratio change +- low-position persistence after transition + +Suggested internal states: + +- `normal` +- `suspicious` +- `active` +- `ended` + +This keeps the data and pipeline correct while allowing replacement by a temporal model later. + +### 6.4 `fight` + +Implementation mode: + +- temporal rule/state-machine in first release + +Decision basis: + +- two or more tracks stay in close proximity +- short-term strong relative motion or repeated box perturbation +- persistence over a short window + +Risk note: + +This event has the highest false-positive risk in a rule-only implementation. First release should be positioned as an early-warning event, not a high-certainty semantic classifier. + +## 7. Event Lifecycle Rules + +All behavior events should use the same lifecycle semantics. + +- `Pending` + - early evidence exists but activation threshold is not yet met +- `Active` + - activation threshold is met and the event should be visible to OSD and alarm +- `Ended` + - event was active and has now stopped, but may still be emitted briefly for lifecycle completion or external consumers + +Core timestamps: + +- `start_pts`: when evidence first appeared +- `last_pts`: latest frame contributing to this event +- `duration_ms`: derived from `start_pts` and `last_pts` + +## 8. Configuration Direction + +### 8.1 `region_event` Example Shape + +```json +{ + "id": "region_evt", + "type": "region_event", + "events": [ + { + "type": "intrusion", + "region_id": "zone_a", + "roi": {"x": 0.1, "y": 0.1, "w": 0.5, "h": 0.5}, + "min_duration_ms": 1000 + }, + { + "type": "climb", + "region_id": "fence_1", + "line": {"x1": 0.2, "y1": 0.4, "x2": 0.8, "y2": 0.4}, + "min_duration_ms": 600 + } + ] +} +``` + +### 8.2 `action_recog` Example Shape + +```json +{ + "id": "action_evt", + "type": "action_recog", + "events": [ + { + "type": "fall", + "window_ms": 1500, + "activate_duration_ms": 500 + }, + { + "type": "fight", + "window_ms": 1200, + "min_tracks": 2, + "activate_duration_ms": 400 + } + ] +} +``` + +These are directional examples, not final schema commitments. Final schema should be kept explicit and event-specific rather than forced into one generic threshold bag. + +## 9. OSD and Alarm Expectations + +### 9.1 OSD + +OSD must be able to render: + +- event label such as `fall`, `fight`, `climb`, `intrusion` +- primary associated track id +- elapsed duration in milliseconds or seconds +- event box if available, otherwise tracked target box + +### 9.2 Alarm + +Alarm rules should support behavior-event inputs directly. Example dimensions: + +- `event_types` +- `region_ids` +- `min_score` +- `min_duration_ms` +- `cooldown_ms` +- `per_track_cooldown_ms` + +The alarm plugin should not reconstruct behavior semantics from detection classes. + +## 10. Non-Goals for First Release + +- full temporal deep-learning action model integration +- skeleton-based action recognition +- multi-camera cross-view identity reasoning +- perfect fight detection accuracy +- replacing existing object detection pipeline + +## 11. Migration and Compatibility + +The design should preserve compatibility with the current object-detection pipeline: + +- existing graphs without behavior nodes must keep working +- `osd` and `alarm` behavior-event support should be additive +- behavior-event support should not require all pipelines to enable it + +## 12. Implementation Order + +Recommended implementation order: + +1. Add behavior event data structures to the frame model +2. Extend `osd` and `alarm` contracts to consume behavior events +3. Implement `region_event` +4. Implement `action_recog` with temporal rule/state-machine version +5. Implement `event_fusion` +6. Add sample graph configs for intrusion, climb, fall, and fight + +## 13. Risks + +- If behavior events are encoded as synthetic detections, downstream semantics will become unstable +- If `fight` is treated as a high-certainty classifier in the first release, user expectations will exceed achievable accuracy +- If event lifecycle is not centralized, `osd` and `alarm` will drift into incompatible interpretations +- If action nodes directly own alert policy, the graph will become harder to tune and reuse + +## 14. Decision Summary + +The project should not implement one generic “action recognition plugin” first. + +It should implement: + +- one dedicated behavior event data model +- one rule-driven region event plugin +- one temporal action recognition plugin +- one event fusion plugin + +This is the shortest path that matches the current project architecture and still supports first-release visibility of event type, associated target, and event duration.