# Region Behavior Recognition Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Add first-release region behavior recognition with unified behavior event data, rule-based region events, temporal-rule action events, event fusion, and downstream OSD/alarm support. **Architecture:** Extend `Frame` with a dedicated behavior-event payload, then add three focused plugins: `region_event`, `action_recog`, and `event_fusion`. Keep the first release linear in the graph so each node appends or normalizes behavior events on the same `Frame`, while `osd` and `alarm` consume the normalized result without overloading detection classes. **Tech Stack:** C++17, existing plugin `INode` ABI, RK3588 DAG graph manager, GoogleTest, CMake --- ## File Map - Modify: `D:\App\C++\Rk3588Sys\include\frame\frame.h` - Add behavior event enums and result structures to `Frame` - Create: `D:\App\C++\Rk3588Sys\include\behavior\behavior_event.h` - Shared behavior-event definitions and helpers if `frame.h` becomes noisy - Modify: `D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt` - Register new plugins in the build - Create: `D:\App\C++\Rk3588Sys\plugins\region_event\region_event_node.cpp` - Rule-based intrusion and climb events - Create: `D:\App\C++\Rk3588Sys\plugins\action_recog\action_recog_node.cpp` - Temporal-rule fall and fight events - Create: `D:\App\C++\Rk3588Sys\plugins\event_fusion\event_fusion_node.cpp` - Stable event normalization, ID assignment, and lifecycle progression - Modify: `D:\App\C++\Rk3588Sys\plugins\osd\osd_node.cpp` - Draw behavior labels, track IDs, and durations - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.h` - Add behavior-event-aware alarm rule shape - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.cpp` - Evaluate behavior events in addition to detections - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\alarm_node.cpp` - Parse behavior-event alarm rules and preserve old detection rules - Create: `D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp` - Validate frame-level behavior event model - Create: `D:\App\C++\Rk3588Sys\tests\test_region_event.cpp` - Validate intrusion and climb rule logic - Create: `D:\App\C++\Rk3588Sys\tests\test_action_recog.cpp` - Validate fall and fight temporal state-machine logic - Create: `D:\App\C++\Rk3588Sys\tests\test_event_fusion.cpp` - Validate event merge and lifecycle behavior - Create: `D:\App\C++\Rk3588Sys\tests\test_alarm_behavior_events.cpp` - Validate alarm matching on behavior events - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - Add new test files to `rk3588_gtests` - Create: `D:\App\C++\Rk3588Sys\configs\sample_region_behavior_intrusion.json` - Reference graph for intrusion pipeline - Create: `D:\App\C++\Rk3588Sys\configs\sample_region_behavior_full.json` - Reference graph for intrusion, climb, fall, and fight - Modify: `D:\App\C++\Rk3588Sys\Readme.md` - Document behavior event model and sample pipeline ### Task 1: Add Behavior Event Data Model **Files:** - Create: `D:\App\C++\Rk3588Sys\include\behavior\behavior_event.h` - Modify: `D:\App\C++\Rk3588Sys\include\frame\frame.h` - Test: `D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp` - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - [ ] **Step 1: Write the failing test** ```cpp #include #include "frame/frame.h" namespace rk3588 { TEST(BehaviorEventModelTest, FrameStoresBehaviorEventsSeparatelyFromDetections) { auto frame = std::make_shared(); frame->det = std::make_shared(); frame->behavior_events = std::make_shared(); BehaviorEventItem item; item.event_id = 7; item.type = BehaviorEventType::Fall; item.status = BehaviorEventStatus::Active; item.track_ids = {42}; item.duration_ms = 1600; frame->behavior_events->items.push_back(item); ASSERT_NE(frame->det, nullptr); ASSERT_NE(frame->behavior_events, nullptr); ASSERT_EQ(frame->behavior_events->items.size(), 1u); EXPECT_EQ(frame->behavior_events->items[0].track_ids[0], 42); EXPECT_EQ(frame->behavior_events->items[0].duration_ms, 1600u); } } // namespace rk3588 ``` - [ ] **Step 2: Run test to verify it fails** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure` Expected: compile failure because `BehaviorEventResult`, `BehaviorEventItem`, and related enums do not exist. - [ ] **Step 3: Write minimal implementation** ```cpp // include/behavior/behavior_event.h #pragma once #include #include #include #include "frame/frame.h" namespace rk3588 { enum class BehaviorEventType { Intrusion, Climb, Fall, Fight }; enum class BehaviorEventStatus { Pending, Active, Ended }; struct BehaviorEventItem { int event_id = -1; BehaviorEventType type = BehaviorEventType::Intrusion; BehaviorEventStatus status = BehaviorEventStatus::Pending; float score = 0.0f; Rect bbox{}; std::vector track_ids; uint64_t start_pts = 0; uint64_t last_pts = 0; uint64_t duration_ms = 0; std::string source; std::string region_id; }; struct BehaviorEventResult { std::vector items; }; } // namespace rk3588 ``` ```cpp // include/frame/frame.h #include "behavior/behavior_event.h" struct Frame { // existing fields... std::shared_ptr behavior_events; }; ``` - [ ] **Step 4: Run test to verify it passes** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure` Expected: `BehaviorEventModelTest` passes. - [ ] **Step 5: Commit** ```bash git add include/behavior/behavior_event.h include/frame/frame.h tests/test_behavior_event_model.cpp tests/CMakeLists.txt git commit -m "feat: add behavior event frame model" ``` ### Task 2: Implement `region_event` for Intrusion and Rule-Based Climb **Files:** - Create: `D:\App\C++\Rk3588Sys\plugins\region_event\region_event_node.cpp` - Modify: `D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt` - Test: `D:\App\C++\Rk3588Sys\tests\test_region_event.cpp` - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - [ ] **Step 1: Write the failing tests** ```cpp #include #include "frame/frame.h" namespace rk3588 { class RegionEventNode; TEST(RegionEventTest, EmitsIntrusionWhenTrackedPersonStaysInsideRegion) { RegionEventNode node; // Build config with one intrusion ROI and min_duration_ms = 0 for deterministic test. // Feed a frame with one tracked person inside the ROI. // Expect one active intrusion event. } TEST(RegionEventTest, EmitsClimbWhenTrackCrossesBoundaryOverTime) { RegionEventNode node; // Build config with one climb boundary. // Feed a short sequence where one track moves across boundary with upward motion. // Expect one climb event with the same track_id. } } // namespace rk3588 ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R RegionEventTest --output-on-failure` Expected: compile failure because `RegionEventNode` and plugin logic do not exist. - [ ] **Step 3: Write minimal implementation** ```cpp // plugins/region_event/region_event_node.cpp class RegionEventNode final : public INode { public: std::string Id() const override { return id_; } std::string Type() const override { return "region_event"; } bool Init(const SimpleJson& config, const NodeContext& ctx) override; bool Start() override { return true; } void Stop() override {} NodeStatus Process(FramePtr frame) override { if (!frame || !frame->det) return NodeStatus::DROP; EnsureBehaviorEvents(*frame); EvaluateIntrusion(*frame); EvaluateClimb(*frame); Push(frame); return NodeStatus::OK; } private: void EnsureBehaviorEvents(Frame& frame); void EvaluateIntrusion(Frame& frame); void EvaluateClimb(Frame& frame); void Push(const FramePtr& frame); std::string id_; NodeContext ctx_{}; }; REGISTER_NODE(RegionEventNode, "region_event"); ``` Implementation constraints: - Only inspect detections with valid `track_id` - Append to `frame->behavior_events` without mutating `frame->det` - Store per-track state inside the node for duration and boundary-crossing checks - [ ] **Step 4: Run tests to verify they pass** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R RegionEventTest --output-on-failure` Expected: intrusion and climb tests pass. - [ ] **Step 5: Commit** ```bash git add plugins/region_event/region_event_node.cpp plugins/CMakeLists.txt tests/test_region_event.cpp tests/CMakeLists.txt git commit -m "feat: add region event plugin" ``` ### Task 3: Implement `action_recog` for Temporal Fall and Fight **Files:** - Create: `D:\App\C++\Rk3588Sys\plugins\action_recog\action_recog_node.cpp` - Modify: `D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt` - Test: `D:\App\C++\Rk3588Sys\tests\test_action_recog.cpp` - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - [ ] **Step 1: Write the failing tests** ```cpp #include namespace rk3588 { class ActionRecogNode; TEST(ActionRecogTest, EmitsFallAfterRapidDropAndLowPosePersistence) { ActionRecogNode node; // Feed same track across frames with centroid drop and aspect-ratio change. // Expect one active fall event. } TEST(ActionRecogTest, EmitsFightWhenTwoTracksStayCloseWithRepeatedMotion) { ActionRecogNode node; // Feed two tracked persons across frames in close proximity with alternating motion. // Expect one active fight event referencing both track ids. } } // namespace rk3588 ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R ActionRecogTest --output-on-failure` Expected: compile failure because `ActionRecogNode` does not exist. - [ ] **Step 3: Write minimal implementation** ```cpp // plugins/action_recog/action_recog_node.cpp struct TrackSample { uint64_t pts = 0; Rect bbox{}; }; class ActionRecogNode final : public INode { public: std::string Id() const override { return id_; } std::string Type() const override { return "action_recog"; } bool Init(const SimpleJson& config, const NodeContext& ctx) override; bool Start() override { return true; } void Stop() override {} NodeStatus Process(FramePtr frame) override; private: void UpdateTrackHistory(const Frame& frame); void EvaluateFall(Frame& frame); void EvaluateFight(Frame& frame); std::string id_; NodeContext ctx_{}; std::unordered_map> history_; }; REGISTER_NODE(ActionRecogNode, "action_recog"); ``` Implementation constraints: - Keep first release rule/state-machine only - Do not add RKNN model coupling in this task - Emit `track_ids` with one track for `fall` and two-or-more tracks for `fight` - [ ] **Step 4: Run tests to verify they pass** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R ActionRecogTest --output-on-failure` Expected: fall and fight tests pass. - [ ] **Step 5: Commit** ```bash git add plugins/action_recog/action_recog_node.cpp plugins/CMakeLists.txt tests/test_action_recog.cpp tests/CMakeLists.txt git commit -m "feat: add temporal action recognition plugin" ``` ### Task 4: Implement `event_fusion` for Event IDs and Lifecycle Normalization **Files:** - Create: `D:\App\C++\Rk3588Sys\plugins\event_fusion\event_fusion_node.cpp` - Modify: `D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt` - Test: `D:\App\C++\Rk3588Sys\tests\test_event_fusion.cpp` - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - [ ] **Step 1: Write the failing tests** ```cpp #include namespace rk3588 { class EventFusionNode; TEST(EventFusionTest, AssignsStableEventIdAcrossFramesForSameTrackAndType) { EventFusionNode node; // Feed two frames with the same fall event on track 42. // Expect the same event_id in both outputs. } TEST(EventFusionTest, MarksEventEndedWhenSourceStopsEmittingIt) { EventFusionNode node; // Feed one frame with an active intrusion event, then one frame without it. // Expect an ended event emission before cleanup. } } // namespace rk3588 ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R EventFusionTest --output-on-failure` Expected: compile failure because `EventFusionNode` does not exist. - [ ] **Step 3: Write minimal implementation** ```cpp // plugins/event_fusion/event_fusion_node.cpp class EventFusionNode final : public INode { public: std::string Id() const override { return id_; } std::string Type() const override { return "event_fusion"; } bool Init(const SimpleJson& config, const NodeContext& ctx) override; bool Start() override { return true; } void Stop() override {} NodeStatus Process(FramePtr frame) override; private: std::string MakeKey(const BehaviorEventItem& item) const; std::string id_; NodeContext ctx_{}; int next_event_id_ = 1; std::unordered_map active_ids_; }; REGISTER_NODE(EventFusionNode, "event_fusion"); ``` Normalization rules: - key identity by `type + region_id + normalized track_ids` - reuse prior `event_id` for matching active events - emit one final `Ended` item before forgetting an event - [ ] **Step 4: Run tests to verify they pass** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R EventFusionTest --output-on-failure` Expected: fusion tests pass. - [ ] **Step 5: Commit** ```bash git add plugins/event_fusion/event_fusion_node.cpp plugins/CMakeLists.txt tests/test_event_fusion.cpp tests/CMakeLists.txt git commit -m "feat: add behavior event fusion plugin" ``` ### Task 5: Update OSD to Render Behavior Events **Files:** - Modify: `D:\App\C++\Rk3588Sys\plugins\osd\osd_node.cpp` - Test: `D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp` - [ ] **Step 1: Write the failing test** ```cpp TEST(BehaviorEventModelTest, BehaviorEventsExposeTypeTrackAndDurationForOsdFormatting) { BehaviorEventItem item; item.type = BehaviorEventType::Intrusion; item.track_ids = {9}; item.duration_ms = 2100; EXPECT_EQ(FormatBehaviorEventLabel(item), "intrusion #9 2.1s"); } ``` - [ ] **Step 2: Run test to verify it fails** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure` Expected: compile failure because behavior-event label formatting helper does not exist. - [ ] **Step 3: Write minimal implementation** ```cpp // plugins/osd/osd_node.cpp static std::string BehaviorEventTypeName(BehaviorEventType type) { switch (type) { case BehaviorEventType::Intrusion: return "intrusion"; case BehaviorEventType::Climb: return "climb"; case BehaviorEventType::Fall: return "fall"; case BehaviorEventType::Fight: return "fight"; } return "unknown"; } static std::string FormatBehaviorEventLabel(const BehaviorEventItem& item) { std::ostringstream oss; oss << BehaviorEventTypeName(item.type); if (!item.track_ids.empty()) oss << " #" << item.track_ids.front(); oss << " " << std::fixed << std::setprecision(1) << (static_cast(item.duration_ms) / 1000.0) << "s"; return oss.str(); } ``` Render rules: - draw behavior boxes after detection boxes - use distinct color per event type - skip `Pending` by default and render `Active` and `Ended` - [ ] **Step 4: Run test to verify it passes** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure` Expected: formatting test passes. - [ ] **Step 5: Commit** ```bash git add plugins/osd/osd_node.cpp tests/test_behavior_event_model.cpp git commit -m "feat: render behavior events in osd" ``` ### Task 6: Extend Alarm Engine to Consume Behavior Events **Files:** - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.h` - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.cpp` - Modify: `D:\App\C++\Rk3588Sys\plugins\alarm\alarm_node.cpp` - Test: `D:\App\C++\Rk3588Sys\tests\test_alarm_behavior_events.cpp` - Modify: `D:\App\C++\Rk3588Sys\tests\CMakeLists.txt` - [ ] **Step 1: Write the failing tests** ```cpp #include namespace rk3588 { TEST(AlarmBehaviorEventsTest, MatchesBehaviorEventByTypeAndDuration) { RuleEngine engine; // Init with one behavior-event rule for fall, min_duration_ms = 1000. // Feed frame with one active fall event of 1500 ms. // Expect one rule match. } TEST(AlarmBehaviorEventsTest, PreservesLegacyDetectionRules) { RuleEngine engine; // Init with a classic detection rule on class_id 0. // Feed frame->det only. // Expect legacy path still matches. } } // namespace rk3588 ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R AlarmBehaviorEventsTest --output-on-failure` Expected: compile failure because rule engine does not understand behavior-event rules. - [ ] **Step 3: Write minimal implementation** ```cpp // plugins/alarm/rule_engine.h struct AlarmRule { std::string name; std::set class_ids; std::set event_types; std::set region_ids; RoiRect roi; float min_score = 0.0f; bool use_behavior_events = false; int min_duration_ms = 0; int cooldown_ms = 5000; }; ``` ```cpp // plugins/alarm/rule_engine.cpp RuleMatchResult RuleEngine::Evaluate(const std::shared_ptr& frame) { if (HasBehaviorRules() && frame && frame->behavior_events) { return EvaluateBehaviorEvents(frame); } return EvaluateDetections(frame); } ``` Compatibility rules: - behavior-event rules must not break existing `class_ids` rules - keep old config shape valid - require explicit `use_behavior_events` or `event_types` to enter new path - [ ] **Step 4: Run tests to verify they pass** Run: `cmake --build build --target rk3588_gtests && ctest --test-dir build -R AlarmBehaviorEventsTest --output-on-failure` Expected: both behavior-event and legacy detection tests pass. - [ ] **Step 5: Commit** ```bash git add plugins/alarm/rule_engine.h plugins/alarm/rule_engine.cpp plugins/alarm/alarm_node.cpp tests/test_alarm_behavior_events.cpp tests/CMakeLists.txt git commit -m "feat: support behavior events in alarm engine" ``` ### Task 7: Add Sample Configs and Documentation **Files:** - Create: `D:\App\C++\Rk3588Sys\configs\sample_region_behavior_intrusion.json` - Create: `D:\App\C++\Rk3588Sys\configs\sample_region_behavior_full.json` - Modify: `D:\App\C++\Rk3588Sys\Readme.md` - [ ] **Step 1: Write the config files** ```json { "queue": {"size": 8, "strategy": "drop_oldest"}, "graphs": [ { "name": "sample_region_behavior_intrusion", "nodes": [ {"id": "in", "type": "input_rtsp", "url": "rtsp://127.0.0.1/live"}, {"id": "pre", "type": "preprocess", "dst_w": 1920, "dst_h": 1080, "dst_format": "rgb"}, {"id": "person_det", "type": "ai_yolo", "model_path": "./models/object_det_yolov8n_coco_640_rk3588.rknn", "model_version": "v8", "model_w": 640, "model_h": 640, "class_filter": [0]}, {"id": "trk", "type": "tracker", "track_classes": [0]}, {"id": "region_evt", "type": "region_event", "events": [{"type": "intrusion", "region_id": "zone_a", "roi": {"x": 0.1, "y": 0.1, "w": 0.4, "h": 0.4}, "min_duration_ms": 1000}]}, {"id": "fusion", "type": "event_fusion"}, {"id": "osd", "type": "osd"}, {"id": "pub", "type": "publish", "outputs": [{"proto": "rtsp_server", "port": 8555, "path": "/live/behavior"}]}, {"id": "alarm", "type": "alarm", "rules": [{"name": "intrusion_rule", "use_behavior_events": true, "event_types": ["intrusion"], "min_duration_ms": 1000, "cooldown_ms": 10000}]} ], "edges": [["in", "pre"], ["pre", "person_det"], ["person_det", "trk"], ["trk", "region_evt"], ["region_evt", "fusion"], ["fusion", "osd"], ["osd", "pub"], ["pub", "alarm"]] } ] } ``` - [ ] **Step 2: Document behavior-event pipeline in README** ```md ## Behavior Events Behavior events are emitted through `frame->behavior_events`, not encoded as detection classes. First-release nodes: - `region_event`: intrusion, rule-based climb - `action_recog`: temporal-rule fall, temporal-rule fight - `event_fusion`: stable event IDs and lifecycle normalization ``` - [ ] **Step 3: Validate config syntax** Run: `cmake --build build --target rk3588_gtests` Expected: project still builds after adding documentation and sample configs. - [ ] **Step 4: Commit** ```bash git add configs/sample_region_behavior_intrusion.json configs/sample_region_behavior_full.json Readme.md git commit -m "docs: add behavior event sample pipelines" ``` ### Task 8: Full Verification Pass **Files:** - Modify as needed based on failures from previous tasks - [ ] **Step 1: Build the full test target** Run: `cmake --build build --target rk3588_gtests` Expected: build succeeds. - [ ] **Step 2: Run targeted behavior-event tests** Run: `ctest --test-dir build -R "BehaviorEventModelTest|RegionEventTest|ActionRecogTest|EventFusionTest|AlarmBehaviorEventsTest" --output-on-failure` Expected: all new behavior-related tests pass. - [ ] **Step 3: Run existing regression tests** Run: `ctest --test-dir build --output-on-failure` Expected: no regressions in existing gtests. - [ ] **Step 4: Smoke-check plugin build outputs** Run: `cmake --build build --target region_event action_recog event_fusion osd alarm` Expected: all plugin targets build successfully. - [ ] **Step 5: Commit any final fixes** ```bash git add -A git commit -m "test: verify region behavior recognition pipeline" ``` ## Self-Review Spec coverage check: - Unified behavior event data model: covered by Task 1 - `region_event`, `action_recog`, `event_fusion`: covered by Tasks 2-4 - OSD behavior-event rendering: covered by Task 5 - Alarm behavior-event support: covered by Task 6 - Sample graph and docs: covered by Task 7 - Verification and regression safety: covered by Task 8 Placeholder scan: - No `TBD`, `TODO`, or “implement later” markers remain - Each task lists concrete files and exact commands - Each code task includes a code sketch to anchor implementation Type consistency check: - Shared naming uses `BehaviorEventType`, `BehaviorEventStatus`, `BehaviorEventItem`, `BehaviorEventResult` - Plugin names align with the approved architecture: `region_event`, `action_recog`, `event_fusion` - Alarm config extension consistently refers to `use_behavior_events`, `event_types`, and `region_ids`