Add region behavior recognition design spec
This commit is contained in:
parent
a5ed861200
commit
b0dd2f5eee
@ -0,0 +1,449 @@
|
||||
# Region Behavior Recognition Design
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Add region behavior recognition to the existing plugin-based DAG pipeline without breaking the current detection-oriented data model.
|
||||
|
||||
The first release must support:
|
||||
|
||||
- Distinguishing event type in OSD and event output
|
||||
- Associating each event with one or more tracked persons
|
||||
- Reporting how long the event has lasted
|
||||
|
||||
Target event types:
|
||||
|
||||
- `intrusion`
|
||||
- `climb`
|
||||
- `fall`
|
||||
- `fight`
|
||||
|
||||
## 2. Problem Framing
|
||||
|
||||
The current project architecture is plugin-based and detection-centric:
|
||||
|
||||
- `tracker` maintains stable `track_id`
|
||||
- AI nodes write object results into `frame->det`
|
||||
- `osd` and `alarm` mainly consume object-like outputs
|
||||
|
||||
This is not sufficient for behavior events because behavior events are not object detections. A behavior event needs:
|
||||
|
||||
- event identity
|
||||
- event type
|
||||
- lifecycle state
|
||||
- associated track ids
|
||||
- duration
|
||||
- optional region context
|
||||
|
||||
Encoding behavior events as synthetic detection classes would create a false model and couple downstream nodes to plugin-specific conventions. The design therefore introduces a dedicated behavior event data structure.
|
||||
|
||||
## 3. Recommended Architecture
|
||||
|
||||
The design separates rule-driven region events from temporal semantic events.
|
||||
|
||||
### 3.1 New and Updated Nodes
|
||||
|
||||
- `tracker`
|
||||
- Existing node
|
||||
- Keeps responsibility limited to stable `track_id` assignment
|
||||
- `region_event`
|
||||
- New plugin
|
||||
- Handles rule-driven region events such as `intrusion` and rule-based `climb`
|
||||
- `action_recog`
|
||||
- New plugin
|
||||
- Handles temporal semantic events such as `fall` and `fight`
|
||||
- `event_fusion`
|
||||
- New plugin
|
||||
- Merges behavior events from upstream nodes into one stable output stream
|
||||
- `osd`
|
||||
- Updated to render behavior events
|
||||
- `alarm`
|
||||
- Updated to evaluate behavior events directly
|
||||
|
||||
### 3.2 First-Release Pipeline Shape
|
||||
|
||||
To minimize graph framework changes in the first iteration, behavior nodes should be chained serially while sharing the same `Frame` object:
|
||||
|
||||
```text
|
||||
input -> preprocess -> person_det -> tracker -> region_event -> action_recog -> event_fusion -> osd -> publish -> alarm
|
||||
```
|
||||
|
||||
This keeps the first release compatible with the current linear graph style while still enabling future branch-and-merge topologies if needed.
|
||||
|
||||
## 4. Data Model
|
||||
|
||||
### 4.1 New Behavior Event Types
|
||||
|
||||
Add a dedicated behavior event result to the frame model.
|
||||
|
||||
```cpp
|
||||
enum class BehaviorEventType {
|
||||
Intrusion,
|
||||
Climb,
|
||||
Fall,
|
||||
Fight
|
||||
};
|
||||
|
||||
enum class BehaviorEventStatus {
|
||||
Pending,
|
||||
Active,
|
||||
Ended
|
||||
};
|
||||
|
||||
struct BehaviorEventItem {
|
||||
int event_id = -1;
|
||||
BehaviorEventType type = BehaviorEventType::Intrusion;
|
||||
BehaviorEventStatus status = BehaviorEventStatus::Pending;
|
||||
|
||||
float score = 0.0f;
|
||||
|
||||
Rect bbox{};
|
||||
std::vector<int> track_ids;
|
||||
|
||||
uint64_t start_pts = 0;
|
||||
uint64_t last_pts = 0;
|
||||
uint64_t duration_ms = 0;
|
||||
|
||||
std::string source;
|
||||
std::string region_id;
|
||||
};
|
||||
|
||||
struct BehaviorEventResult {
|
||||
std::vector<BehaviorEventItem> items;
|
||||
};
|
||||
```
|
||||
|
||||
### 4.2 Frame Extension
|
||||
|
||||
Add this field to `Frame`:
|
||||
|
||||
```cpp
|
||||
std::shared_ptr<BehaviorEventResult> behavior_events;
|
||||
```
|
||||
|
||||
### 4.3 Design Rationale
|
||||
|
||||
This preserves a clean separation:
|
||||
|
||||
- `det` describes detected objects
|
||||
- `behavior_events` describes interpreted events
|
||||
|
||||
This avoids overloading `cls_id`, avoids hidden `user_meta` contracts, and provides one stable contract for `event_fusion`, `osd`, `alarm`, and future APIs.
|
||||
|
||||
## 5. Node Responsibilities
|
||||
|
||||
### 5.1 `tracker`
|
||||
|
||||
Input:
|
||||
|
||||
- person detections in `frame->det`
|
||||
|
||||
Output:
|
||||
|
||||
- updated `frame->det` with stable `track_id`
|
||||
|
||||
Non-goals:
|
||||
|
||||
- no event interpretation
|
||||
- no region semantics
|
||||
|
||||
### 5.2 `region_event`
|
||||
|
||||
Input:
|
||||
|
||||
- `frame->det`
|
||||
- track ids from `tracker`
|
||||
- configured regions and event rules
|
||||
|
||||
Output:
|
||||
|
||||
- appends region-derived events to `frame->behavior_events`
|
||||
|
||||
First-release responsibilities:
|
||||
|
||||
- `intrusion`
|
||||
- rule-based `climb`
|
||||
|
||||
### 5.3 `action_recog`
|
||||
|
||||
Input:
|
||||
|
||||
- tracked targets from `frame->det`
|
||||
- temporal history per `track_id`
|
||||
- optional cropped target imagery or derived temporal features
|
||||
|
||||
Output:
|
||||
|
||||
- appends temporal behavior events to `frame->behavior_events`
|
||||
|
||||
First-release responsibilities:
|
||||
|
||||
- temporal-rule `fall`
|
||||
- temporal-rule `fight`
|
||||
|
||||
Internal requirement:
|
||||
|
||||
- maintain per-track time windows and state machines
|
||||
|
||||
### 5.4 `event_fusion`
|
||||
|
||||
Input:
|
||||
|
||||
- behavior events produced by upstream behavior nodes
|
||||
|
||||
Output:
|
||||
|
||||
- normalized `frame->behavior_events`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- assign and maintain stable `event_id`
|
||||
- merge duplicates from different behavior sources
|
||||
- advance event lifecycle state
|
||||
- keep one downstream-facing event contract
|
||||
|
||||
### 5.5 `osd`
|
||||
|
||||
Input:
|
||||
|
||||
- `frame->behavior_events`
|
||||
|
||||
Output:
|
||||
|
||||
- rendered overlays showing:
|
||||
- event type
|
||||
- associated track id or ids
|
||||
- duration
|
||||
- event box or associated target box
|
||||
|
||||
### 5.6 `alarm`
|
||||
|
||||
Input:
|
||||
|
||||
- `frame->behavior_events`
|
||||
|
||||
Output:
|
||||
|
||||
- event-driven alert actions
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- rule matching by behavior type
|
||||
- cooldown and suppression
|
||||
- duration thresholds
|
||||
- optional region filtering
|
||||
|
||||
## 6. First-Release Event Strategy
|
||||
|
||||
The first release should prioritize correctness of architecture and event lifecycle over ambitious model accuracy.
|
||||
|
||||
### 6.1 `intrusion`
|
||||
|
||||
Implementation mode:
|
||||
|
||||
- rule-based
|
||||
|
||||
Decision basis:
|
||||
|
||||
- tracked target enters configured ROI
|
||||
- target remains inside for at least `min_duration_ms`
|
||||
|
||||
Optional filters:
|
||||
|
||||
- minimum object size
|
||||
- class whitelist
|
||||
- direction gate
|
||||
- confidence threshold
|
||||
|
||||
### 6.2 `climb`
|
||||
|
||||
Implementation mode:
|
||||
|
||||
- rule-based in first release
|
||||
|
||||
Decision basis:
|
||||
|
||||
- tracked target approaches configured climb boundary
|
||||
- target top, center, or bottom crosses boundary over time
|
||||
- target exhibits meaningful vertical motion near the boundary
|
||||
|
||||
Optional filters:
|
||||
|
||||
- boundary side
|
||||
- crossing ratio
|
||||
- dwell time near boundary
|
||||
|
||||
This is intentionally framed as a region-crossing event first, not a semantic action model.
|
||||
|
||||
### 6.3 `fall`
|
||||
|
||||
Implementation mode:
|
||||
|
||||
- temporal rule/state-machine in first release
|
||||
|
||||
Decision basis:
|
||||
|
||||
- rapid centroid drop
|
||||
- significant aspect-ratio change
|
||||
- low-position persistence after transition
|
||||
|
||||
Suggested internal states:
|
||||
|
||||
- `normal`
|
||||
- `suspicious`
|
||||
- `active`
|
||||
- `ended`
|
||||
|
||||
This keeps the data and pipeline correct while allowing replacement by a temporal model later.
|
||||
|
||||
### 6.4 `fight`
|
||||
|
||||
Implementation mode:
|
||||
|
||||
- temporal rule/state-machine in first release
|
||||
|
||||
Decision basis:
|
||||
|
||||
- two or more tracks stay in close proximity
|
||||
- short-term strong relative motion or repeated box perturbation
|
||||
- persistence over a short window
|
||||
|
||||
Risk note:
|
||||
|
||||
This event has the highest false-positive risk in a rule-only implementation. First release should be positioned as an early-warning event, not a high-certainty semantic classifier.
|
||||
|
||||
## 7. Event Lifecycle Rules
|
||||
|
||||
All behavior events should use the same lifecycle semantics.
|
||||
|
||||
- `Pending`
|
||||
- early evidence exists but activation threshold is not yet met
|
||||
- `Active`
|
||||
- activation threshold is met and the event should be visible to OSD and alarm
|
||||
- `Ended`
|
||||
- event was active and has now stopped, but may still be emitted briefly for lifecycle completion or external consumers
|
||||
|
||||
Core timestamps:
|
||||
|
||||
- `start_pts`: when evidence first appeared
|
||||
- `last_pts`: latest frame contributing to this event
|
||||
- `duration_ms`: derived from `start_pts` and `last_pts`
|
||||
|
||||
## 8. Configuration Direction
|
||||
|
||||
### 8.1 `region_event` Example Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "region_evt",
|
||||
"type": "region_event",
|
||||
"events": [
|
||||
{
|
||||
"type": "intrusion",
|
||||
"region_id": "zone_a",
|
||||
"roi": {"x": 0.1, "y": 0.1, "w": 0.5, "h": 0.5},
|
||||
"min_duration_ms": 1000
|
||||
},
|
||||
{
|
||||
"type": "climb",
|
||||
"region_id": "fence_1",
|
||||
"line": {"x1": 0.2, "y1": 0.4, "x2": 0.8, "y2": 0.4},
|
||||
"min_duration_ms": 600
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 8.2 `action_recog` Example Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "action_evt",
|
||||
"type": "action_recog",
|
||||
"events": [
|
||||
{
|
||||
"type": "fall",
|
||||
"window_ms": 1500,
|
||||
"activate_duration_ms": 500
|
||||
},
|
||||
{
|
||||
"type": "fight",
|
||||
"window_ms": 1200,
|
||||
"min_tracks": 2,
|
||||
"activate_duration_ms": 400
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
These are directional examples, not final schema commitments. Final schema should be kept explicit and event-specific rather than forced into one generic threshold bag.
|
||||
|
||||
## 9. OSD and Alarm Expectations
|
||||
|
||||
### 9.1 OSD
|
||||
|
||||
OSD must be able to render:
|
||||
|
||||
- event label such as `fall`, `fight`, `climb`, `intrusion`
|
||||
- primary associated track id
|
||||
- elapsed duration in milliseconds or seconds
|
||||
- event box if available, otherwise tracked target box
|
||||
|
||||
### 9.2 Alarm
|
||||
|
||||
Alarm rules should support behavior-event inputs directly. Example dimensions:
|
||||
|
||||
- `event_types`
|
||||
- `region_ids`
|
||||
- `min_score`
|
||||
- `min_duration_ms`
|
||||
- `cooldown_ms`
|
||||
- `per_track_cooldown_ms`
|
||||
|
||||
The alarm plugin should not reconstruct behavior semantics from detection classes.
|
||||
|
||||
## 10. Non-Goals for First Release
|
||||
|
||||
- full temporal deep-learning action model integration
|
||||
- skeleton-based action recognition
|
||||
- multi-camera cross-view identity reasoning
|
||||
- perfect fight detection accuracy
|
||||
- replacing existing object detection pipeline
|
||||
|
||||
## 11. Migration and Compatibility
|
||||
|
||||
The design should preserve compatibility with the current object-detection pipeline:
|
||||
|
||||
- existing graphs without behavior nodes must keep working
|
||||
- `osd` and `alarm` behavior-event support should be additive
|
||||
- behavior-event support should not require all pipelines to enable it
|
||||
|
||||
## 12. Implementation Order
|
||||
|
||||
Recommended implementation order:
|
||||
|
||||
1. Add behavior event data structures to the frame model
|
||||
2. Extend `osd` and `alarm` contracts to consume behavior events
|
||||
3. Implement `region_event`
|
||||
4. Implement `action_recog` with temporal rule/state-machine version
|
||||
5. Implement `event_fusion`
|
||||
6. Add sample graph configs for intrusion, climb, fall, and fight
|
||||
|
||||
## 13. Risks
|
||||
|
||||
- If behavior events are encoded as synthetic detections, downstream semantics will become unstable
|
||||
- If `fight` is treated as a high-certainty classifier in the first release, user expectations will exceed achievable accuracy
|
||||
- If event lifecycle is not centralized, `osd` and `alarm` will drift into incompatible interpretations
|
||||
- If action nodes directly own alert policy, the graph will become harder to tune and reuse
|
||||
|
||||
## 14. Decision Summary
|
||||
|
||||
The project should not implement one generic “action recognition plugin” first.
|
||||
|
||||
It should implement:
|
||||
|
||||
- one dedicated behavior event data model
|
||||
- one rule-driven region event plugin
|
||||
- one temporal action recognition plugin
|
||||
- one event fusion plugin
|
||||
|
||||
This is the shortest path that matches the current project architecture and still supports first-release visibility of event type, associated target, and event duration.
|
||||
Loading…
Reference in New Issue
Block a user