sladro 2a0a1470cb Add agent guidance and plan scaffolding

2026-04-01 09:59:43 +08:00

23 KiB

Raw Blame History

Region Behavior Recognition Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add first-release region behavior recognition with unified behavior event data, rule-based region events, temporal-rule action events, event fusion, and downstream OSD/alarm support.

Architecture: Extend Frame with a dedicated behavior-event payload, then add three focused plugins: region_event, action_recog, and event_fusion. Keep the first release linear in the graph so each node appends or normalizes behavior events on the same Frame, while osd and alarm consume the normalized result without overloading detection classes.

Tech Stack: C++17, existing plugin INode ABI, RK3588 DAG graph manager, GoogleTest, CMake

File Map

Modify: D:\App\C++\Rk3588Sys\include\frame\frame.h
- Add behavior event enums and result structures to Frame
Create: D:\App\C++\Rk3588Sys\include\behavior\behavior_event.h
- Shared behavior-event definitions and helpers if frame.h becomes noisy
Modify: D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt
- Register new plugins in the build
Create: D:\App\C++\Rk3588Sys\plugins\region_event\region_event_node.cpp
- Rule-based intrusion and climb events
Create: D:\App\C++\Rk3588Sys\plugins\action_recog\action_recog_node.cpp
- Temporal-rule fall and fight events
Create: D:\App\C++\Rk3588Sys\plugins\event_fusion\event_fusion_node.cpp
- Stable event normalization, ID assignment, and lifecycle progression
Modify: D:\App\C++\Rk3588Sys\plugins\osd\osd_node.cpp
- Draw behavior labels, track IDs, and durations
Modify: D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.h
- Add behavior-event-aware alarm rule shape
Modify: D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.cpp
- Evaluate behavior events in addition to detections
Modify: D:\App\C++\Rk3588Sys\plugins\alarm\alarm_node.cpp
- Parse behavior-event alarm rules and preserve old detection rules
Create: D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp
- Validate frame-level behavior event model
Create: D:\App\C++\Rk3588Sys\tests\test_region_event.cpp
- Validate intrusion and climb rule logic
Create: D:\App\C++\Rk3588Sys\tests\test_action_recog.cpp
- Validate fall and fight temporal state-machine logic
Create: D:\App\C++\Rk3588Sys\tests\test_event_fusion.cpp
- Validate event merge and lifecycle behavior
Create: D:\App\C++\Rk3588Sys\tests\test_alarm_behavior_events.cpp
- Validate alarm matching on behavior events
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
- Add new test files to rk3588_gtests
Create: D:\App\C++\Rk3588Sys\configs\sample_region_behavior_intrusion.json
- Reference graph for intrusion pipeline
Create: D:\App\C++\Rk3588Sys\configs\sample_region_behavior_full.json
- Reference graph for intrusion, climb, fall, and fight
Modify: D:\App\C++\Rk3588Sys\Readme.md
- Document behavior event model and sample pipeline

Task 1: Add Behavior Event Data Model

Files:

Create: D:\App\C++\Rk3588Sys\include\behavior\behavior_event.h
Modify: D:\App\C++\Rk3588Sys\include\frame\frame.h
Test: D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
Step 1: Write the failing test

#include <gtest/gtest.h>

#include "frame/frame.h"

namespace rk3588 {

TEST(BehaviorEventModelTest, FrameStoresBehaviorEventsSeparatelyFromDetections) {
    auto frame = std::make_shared<Frame>();
    frame->det = std::make_shared<DetectionResult>();
    frame->behavior_events = std::make_shared<BehaviorEventResult>();

    BehaviorEventItem item;
    item.event_id = 7;
    item.type = BehaviorEventType::Fall;
    item.status = BehaviorEventStatus::Active;
    item.track_ids = {42};
    item.duration_ms = 1600;

    frame->behavior_events->items.push_back(item);

    ASSERT_NE(frame->det, nullptr);
    ASSERT_NE(frame->behavior_events, nullptr);
    ASSERT_EQ(frame->behavior_events->items.size(), 1u);
    EXPECT_EQ(frame->behavior_events->items[0].track_ids[0], 42);
    EXPECT_EQ(frame->behavior_events->items[0].duration_ms, 1600u);
}

}  // namespace rk3588

Step 2: Run test to verify it fails

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure

Expected: compile failure because BehaviorEventResult, BehaviorEventItem, and related enums do not exist.

Step 3: Write minimal implementation

// include/behavior/behavior_event.h
#pragma once

#include <cstdint>
#include <string>
#include <vector>

#include "frame/frame.h"

namespace rk3588 {

enum class BehaviorEventType { Intrusion, Climb, Fall, Fight };
enum class BehaviorEventStatus { Pending, Active, Ended };

struct BehaviorEventItem {
    int event_id = -1;
    BehaviorEventType type = BehaviorEventType::Intrusion;
    BehaviorEventStatus status = BehaviorEventStatus::Pending;
    float score = 0.0f;
    Rect bbox{};
    std::vector<int> track_ids;
    uint64_t start_pts = 0;
    uint64_t last_pts = 0;
    uint64_t duration_ms = 0;
    std::string source;
    std::string region_id;
};

struct BehaviorEventResult {
    std::vector<BehaviorEventItem> items;
};

}  // namespace rk3588

// include/frame/frame.h
#include "behavior/behavior_event.h"

struct Frame {
    // existing fields...
    std::shared_ptr<BehaviorEventResult> behavior_events;
};

Step 4: Run test to verify it passes

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure

Expected: BehaviorEventModelTest passes.

Step 5: Commit

git add include/behavior/behavior_event.h include/frame/frame.h tests/test_behavior_event_model.cpp tests/CMakeLists.txt
git commit -m "feat: add behavior event frame model"

Task 2: Implement `region_event` for Intrusion and Rule-Based Climb

Files:

Create: D:\App\C++\Rk3588Sys\plugins\region_event\region_event_node.cpp
Modify: D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt
Test: D:\App\C++\Rk3588Sys\tests\test_region_event.cpp
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
Step 1: Write the failing tests

#include <gtest/gtest.h>

#include "frame/frame.h"

namespace rk3588 {

class RegionEventNode;

TEST(RegionEventTest, EmitsIntrusionWhenTrackedPersonStaysInsideRegion) {
    RegionEventNode node;
    // Build config with one intrusion ROI and min_duration_ms = 0 for deterministic test.
    // Feed a frame with one tracked person inside the ROI.
    // Expect one active intrusion event.
}

TEST(RegionEventTest, EmitsClimbWhenTrackCrossesBoundaryOverTime) {
    RegionEventNode node;
    // Build config with one climb boundary.
    // Feed a short sequence where one track moves across boundary with upward motion.
    // Expect one climb event with the same track_id.
}

}  // namespace rk3588

Step 2: Run tests to verify they fail

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R RegionEventTest --output-on-failure

Expected: compile failure because RegionEventNode and plugin logic do not exist.

Step 3: Write minimal implementation

// plugins/region_event/region_event_node.cpp
class RegionEventNode final : public INode {
public:
    std::string Id() const override { return id_; }
    std::string Type() const override { return "region_event"; }

    bool Init(const SimpleJson& config, const NodeContext& ctx) override;
    bool Start() override { return true; }
    void Stop() override {}
    NodeStatus Process(FramePtr frame) override {
        if (!frame || !frame->det) return NodeStatus::DROP;
        EnsureBehaviorEvents(*frame);
        EvaluateIntrusion(*frame);
        EvaluateClimb(*frame);
        Push(frame);
        return NodeStatus::OK;
    }

private:
    void EnsureBehaviorEvents(Frame& frame);
    void EvaluateIntrusion(Frame& frame);
    void EvaluateClimb(Frame& frame);
    void Push(const FramePtr& frame);

    std::string id_;
    NodeContext ctx_{};
};

REGISTER_NODE(RegionEventNode, "region_event");

Implementation constraints:

Only inspect detections with valid track_id
Append to frame->behavior_events without mutating frame->det
Store per-track state inside the node for duration and boundary-crossing checks
Step 4: Run tests to verify they pass

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R RegionEventTest --output-on-failure

Expected: intrusion and climb tests pass.

Step 5: Commit

git add plugins/region_event/region_event_node.cpp plugins/CMakeLists.txt tests/test_region_event.cpp tests/CMakeLists.txt
git commit -m "feat: add region event plugin"

Task 3: Implement `action_recog` for Temporal Fall and Fight

Files:

Create: D:\App\C++\Rk3588Sys\plugins\action_recog\action_recog_node.cpp
Modify: D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt
Test: D:\App\C++\Rk3588Sys\tests\test_action_recog.cpp
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
Step 1: Write the failing tests

#include <gtest/gtest.h>

namespace rk3588 {

class ActionRecogNode;

TEST(ActionRecogTest, EmitsFallAfterRapidDropAndLowPosePersistence) {
    ActionRecogNode node;
    // Feed same track across frames with centroid drop and aspect-ratio change.
    // Expect one active fall event.
}

TEST(ActionRecogTest, EmitsFightWhenTwoTracksStayCloseWithRepeatedMotion) {
    ActionRecogNode node;
    // Feed two tracked persons across frames in close proximity with alternating motion.
    // Expect one active fight event referencing both track ids.
}

}  // namespace rk3588

Step 2: Run tests to verify they fail

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R ActionRecogTest --output-on-failure

Expected: compile failure because ActionRecogNode does not exist.

Step 3: Write minimal implementation

// plugins/action_recog/action_recog_node.cpp
struct TrackSample {
    uint64_t pts = 0;
    Rect bbox{};
};

class ActionRecogNode final : public INode {
public:
    std::string Id() const override { return id_; }
    std::string Type() const override { return "action_recog"; }
    bool Init(const SimpleJson& config, const NodeContext& ctx) override;
    bool Start() override { return true; }
    void Stop() override {}
    NodeStatus Process(FramePtr frame) override;

private:
    void UpdateTrackHistory(const Frame& frame);
    void EvaluateFall(Frame& frame);
    void EvaluateFight(Frame& frame);

    std::string id_;
    NodeContext ctx_{};
    std::unordered_map<int, std::deque<TrackSample>> history_;
};

REGISTER_NODE(ActionRecogNode, "action_recog");

Implementation constraints:

Keep first release rule/state-machine only
Do not add RKNN model coupling in this task
Emit track_ids with one track for fall and two-or-more tracks for fight
Step 4: Run tests to verify they pass

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R ActionRecogTest --output-on-failure

Expected: fall and fight tests pass.

Step 5: Commit

git add plugins/action_recog/action_recog_node.cpp plugins/CMakeLists.txt tests/test_action_recog.cpp tests/CMakeLists.txt
git commit -m "feat: add temporal action recognition plugin"

Task 4: Implement `event_fusion` for Event IDs and Lifecycle Normalization

Files:

Create: D:\App\C++\Rk3588Sys\plugins\event_fusion\event_fusion_node.cpp
Modify: D:\App\C++\Rk3588Sys\plugins\CMakeLists.txt
Test: D:\App\C++\Rk3588Sys\tests\test_event_fusion.cpp
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
Step 1: Write the failing tests

#include <gtest/gtest.h>

namespace rk3588 {

class EventFusionNode;

TEST(EventFusionTest, AssignsStableEventIdAcrossFramesForSameTrackAndType) {
    EventFusionNode node;
    // Feed two frames with the same fall event on track 42.
    // Expect the same event_id in both outputs.
}

TEST(EventFusionTest, MarksEventEndedWhenSourceStopsEmittingIt) {
    EventFusionNode node;
    // Feed one frame with an active intrusion event, then one frame without it.
    // Expect an ended event emission before cleanup.
}

}  // namespace rk3588

Step 2: Run tests to verify they fail

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R EventFusionTest --output-on-failure

Expected: compile failure because EventFusionNode does not exist.

Step 3: Write minimal implementation

// plugins/event_fusion/event_fusion_node.cpp
class EventFusionNode final : public INode {
public:
    std::string Id() const override { return id_; }
    std::string Type() const override { return "event_fusion"; }
    bool Init(const SimpleJson& config, const NodeContext& ctx) override;
    bool Start() override { return true; }
    void Stop() override {}
    NodeStatus Process(FramePtr frame) override;

private:
    std::string MakeKey(const BehaviorEventItem& item) const;

    std::string id_;
    NodeContext ctx_{};
    int next_event_id_ = 1;
    std::unordered_map<std::string, int> active_ids_;
};

REGISTER_NODE(EventFusionNode, "event_fusion");

Normalization rules:

key identity by type + region_id + normalized track_ids
reuse prior event_id for matching active events
emit one final Ended item before forgetting an event
Step 4: Run tests to verify they pass

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R EventFusionTest --output-on-failure

Expected: fusion tests pass.

Step 5: Commit

git add plugins/event_fusion/event_fusion_node.cpp plugins/CMakeLists.txt tests/test_event_fusion.cpp tests/CMakeLists.txt
git commit -m "feat: add behavior event fusion plugin"

Task 5: Update OSD to Render Behavior Events

Files:

Modify: D:\App\C++\Rk3588Sys\plugins\osd\osd_node.cpp
Test: D:\App\C++\Rk3588Sys\tests\test_behavior_event_model.cpp
Step 1: Write the failing test

TEST(BehaviorEventModelTest, BehaviorEventsExposeTypeTrackAndDurationForOsdFormatting) {
    BehaviorEventItem item;
    item.type = BehaviorEventType::Intrusion;
    item.track_ids = {9};
    item.duration_ms = 2100;

    EXPECT_EQ(FormatBehaviorEventLabel(item), "intrusion #9 2.1s");
}

Step 2: Run test to verify it fails

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure

Expected: compile failure because behavior-event label formatting helper does not exist.

Step 3: Write minimal implementation

// plugins/osd/osd_node.cpp
static std::string BehaviorEventTypeName(BehaviorEventType type) {
    switch (type) {
        case BehaviorEventType::Intrusion: return "intrusion";
        case BehaviorEventType::Climb: return "climb";
        case BehaviorEventType::Fall: return "fall";
        case BehaviorEventType::Fight: return "fight";
    }
    return "unknown";
}

static std::string FormatBehaviorEventLabel(const BehaviorEventItem& item) {
    std::ostringstream oss;
    oss << BehaviorEventTypeName(item.type);
    if (!item.track_ids.empty()) oss << " #" << item.track_ids.front();
    oss << " " << std::fixed << std::setprecision(1)
        << (static_cast<double>(item.duration_ms) / 1000.0) << "s";
    return oss.str();
}

Render rules:

draw behavior boxes after detection boxes
use distinct color per event type
skip Pending by default and render Active and Ended
Step 4: Run test to verify it passes

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R BehaviorEventModelTest --output-on-failure

Expected: formatting test passes.

Step 5: Commit

git add plugins/osd/osd_node.cpp tests/test_behavior_event_model.cpp
git commit -m "feat: render behavior events in osd"

Task 6: Extend Alarm Engine to Consume Behavior Events

Files:

Modify: D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.h
Modify: D:\App\C++\Rk3588Sys\plugins\alarm\rule_engine.cpp
Modify: D:\App\C++\Rk3588Sys\plugins\alarm\alarm_node.cpp
Test: D:\App\C++\Rk3588Sys\tests\test_alarm_behavior_events.cpp
Modify: D:\App\C++\Rk3588Sys\tests\CMakeLists.txt
Step 1: Write the failing tests

#include <gtest/gtest.h>

namespace rk3588 {

TEST(AlarmBehaviorEventsTest, MatchesBehaviorEventByTypeAndDuration) {
    RuleEngine engine;
    // Init with one behavior-event rule for fall, min_duration_ms = 1000.
    // Feed frame with one active fall event of 1500 ms.
    // Expect one rule match.
}

TEST(AlarmBehaviorEventsTest, PreservesLegacyDetectionRules) {
    RuleEngine engine;
    // Init with a classic detection rule on class_id 0.
    // Feed frame->det only.
    // Expect legacy path still matches.
}

}  // namespace rk3588

Step 2: Run tests to verify they fail

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R AlarmBehaviorEventsTest --output-on-failure

Expected: compile failure because rule engine does not understand behavior-event rules.

Step 3: Write minimal implementation

// plugins/alarm/rule_engine.h
struct AlarmRule {
    std::string name;
    std::set<int> class_ids;
    std::set<std::string> event_types;
    std::set<std::string> region_ids;
    RoiRect roi;
    float min_score = 0.0f;
    bool use_behavior_events = false;
    int min_duration_ms = 0;
    int cooldown_ms = 5000;
};

// plugins/alarm/rule_engine.cpp
RuleMatchResult RuleEngine::Evaluate(const std::shared_ptr<Frame>& frame) {
    if (HasBehaviorRules() && frame && frame->behavior_events) {
        return EvaluateBehaviorEvents(frame);
    }
    return EvaluateDetections(frame);
}

Compatibility rules:

behavior-event rules must not break existing class_ids rules
keep old config shape valid
require explicit use_behavior_events or event_types to enter new path
Step 4: Run tests to verify they pass

Run: cmake --build build --target rk3588_gtests && ctest --test-dir build -R AlarmBehaviorEventsTest --output-on-failure

Expected: both behavior-event and legacy detection tests pass.

Step 5: Commit

git add plugins/alarm/rule_engine.h plugins/alarm/rule_engine.cpp plugins/alarm/alarm_node.cpp tests/test_alarm_behavior_events.cpp tests/CMakeLists.txt
git commit -m "feat: support behavior events in alarm engine"

Task 7: Add Sample Configs and Documentation

Files:

Create: D:\App\C++\Rk3588Sys\configs\sample_region_behavior_intrusion.json
Create: D:\App\C++\Rk3588Sys\configs\sample_region_behavior_full.json
Modify: D:\App\C++\Rk3588Sys\Readme.md
Step 1: Write the config files

{
  "queue": {"size": 8, "strategy": "drop_oldest"},
  "graphs": [
    {
      "name": "sample_region_behavior_intrusion",
      "nodes": [
        {"id": "in", "type": "input_rtsp", "url": "rtsp://127.0.0.1/live"},
        {"id": "pre", "type": "preprocess", "dst_w": 1920, "dst_h": 1080, "dst_format": "rgb"},
        {"id": "person_det", "type": "ai_yolo", "model_path": "./models/yolov8n-640.rknn", "model_version": "v8", "model_w": 640, "model_h": 640, "class_filter": [0]},
        {"id": "trk", "type": "tracker", "track_classes": [0]},
        {"id": "region_evt", "type": "region_event", "events": [{"type": "intrusion", "region_id": "zone_a", "roi": {"x": 0.1, "y": 0.1, "w": 0.4, "h": 0.4}, "min_duration_ms": 1000}]},
        {"id": "fusion", "type": "event_fusion"},
        {"id": "osd", "type": "osd"},
        {"id": "pub", "type": "publish", "outputs": [{"proto": "rtsp_server", "port": 8555, "path": "/live/behavior"}]},
        {"id": "alarm", "type": "alarm", "rules": [{"name": "intrusion_rule", "use_behavior_events": true, "event_types": ["intrusion"], "min_duration_ms": 1000, "cooldown_ms": 10000}]}
      ],
      "edges": [["in", "pre"], ["pre", "person_det"], ["person_det", "trk"], ["trk", "region_evt"], ["region_evt", "fusion"], ["fusion", "osd"], ["osd", "pub"], ["pub", "alarm"]]
    }
  ]
}

Step 2: Document behavior-event pipeline in README

## Behavior Events

Behavior events are emitted through `frame->behavior_events`, not encoded as detection classes.

First-release nodes:

- `region_event`: intrusion, rule-based climb
- `action_recog`: temporal-rule fall, temporal-rule fight
- `event_fusion`: stable event IDs and lifecycle normalization

Step 3: Validate config syntax

Run: cmake --build build --target rk3588_gtests

Expected: project still builds after adding documentation and sample configs.

Step 4: Commit

git add configs/sample_region_behavior_intrusion.json configs/sample_region_behavior_full.json Readme.md
git commit -m "docs: add behavior event sample pipelines"

Task 8: Full Verification Pass

Files:

Modify as needed based on failures from previous tasks
Step 1: Build the full test target

Run: cmake --build build --target rk3588_gtests

Expected: build succeeds.

Step 2: Run targeted behavior-event tests

Run: ctest --test-dir build -R "BehaviorEventModelTest|RegionEventTest|ActionRecogTest|EventFusionTest|AlarmBehaviorEventsTest" --output-on-failure

Expected: all new behavior-related tests pass.

Step 3: Run existing regression tests

Run: ctest --test-dir build --output-on-failure

Expected: no regressions in existing gtests.

Step 4: Smoke-check plugin build outputs

Run: cmake --build build --target region_event action_recog event_fusion osd alarm

Expected: all plugin targets build successfully.

Step 5: Commit any final fixes

git add -A
git commit -m "test: verify region behavior recognition pipeline"

Self-Review

Spec coverage check:

Unified behavior event data model: covered by Task 1
region_event, action_recog, event_fusion: covered by Tasks 2-4
OSD behavior-event rendering: covered by Task 5
Alarm behavior-event support: covered by Task 6
Sample graph and docs: covered by Task 7
Verification and regression safety: covered by Task 8

Placeholder scan:

No TBD, TODO, or “implement later” markers remain
Each task lists concrete files and exact commands
Each code task includes a code sketch to anchor implementation

Type consistency check:

Shared naming uses BehaviorEventType, BehaviorEventStatus, BehaviorEventItem, BehaviorEventResult
Plugin names align with the approved architecture: region_event, action_recog, event_fusion
Alarm config extension consistently refers to use_behavior_events, event_types, and region_ids

23 KiB Raw Blame History

Region Behavior Recognition Implementation Plan

File Map

Task 1: Add Behavior Event Data Model

Task 2: Implement region_event for Intrusion and Rule-Based Climb

Task 3: Implement action_recog for Temporal Fall and Fight

Task 4: Implement event_fusion for Event IDs and Lifecycle Normalization

Task 5: Update OSD to Render Behavior Events

Task 6: Extend Alarm Engine to Consume Behavior Events

Task 7: Add Sample Configs and Documentation

Task 8: Full Verification Pass

Self-Review

23 KiB

Raw Blame History

Task 2: Implement `region_event` for Intrusion and Rule-Based Climb

Task 3: Implement `action_recog` for Temporal Fall and Fight

Task 4: Implement `event_fusion` for Event IDs and Lifecycle Normalization