I have been working on a research project called Organic Intelligence Protocol, or OIP.

The idea started with a question I kept coming back to. Are we evaluating intelligence fairly when access to infrastructure and resources can influence how capable a system appears to be?

OIP is my attempt to explore that question through concepts such as infrastructure independence, grounded feedback, and adaptive moral logic.

Over the past few months, I have been documenting the framework, reviewing its mathematical definitions, and auditing its use with four datasets from Bangladesh, Malawi, Uganda, and Ethiopia.

The results have been more useful for identifying limitations than for making big claims.

The audits did not establish enough evidence to authorize the constructs for further empirical evaluation. Some definitions still need clearer measurement rules, and a variable that looks relevant on paper cannot automatically be treated as a valid measurement.

That has changed how I approach the project. Instead of forcing a score or trying to make the framework look complete,, I am keeping unresolved questions open and documenting what the available evidence can and cannot support.

OIP is still research in progress. I am not claiming that it is validated or better than existing approaches.

I would like to hear from people who have worked on AI evaluation, measurement theory, or empirical research.

What would you want to see before taking a framework like this seriously? Where do you think the biggest methodological problems might be?

Critical feedback is welcome. I would rather identify a fundamental problem now than build further work on an assumption that has not been tested.

I'm an undergraduate researcher working independently on conversational memory systems. I developed LSREP to evaluate how persistent memory accumulates, changes and retrieves information over the course of an evolving conversation.

Instead of relying only on isolated endpoint questions, LSREP combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks.

The accompanying architectural case study is ICE v2, a local-first conversational memory system.

The evaluation covers 1,985 turns, 219 distinct probes and 1,211 probe-checkpoint observations across 52 checkpoints, using a private single-user history.

An important part of the findings is that ICE v2 did not outperform pure vector-RAG on the matched LongMemEval evaluation. The ordinary-density replay results also did not establish a quality advantage. The paper discusses the limitations and mechanism-fidelity findings rather than treating the architecture as validated in its entirety.

I'm particularly interested in feedback on the evaluation methodology: what else should a longitudinal memory benchmark test to distinguish successful recall from reliable maintenance of evolving conversational state?

Paper: https://arxiv.org/abs/2609.16730

Code and evaluation artifacts: https://github.com/Deepnar/ice

Frozen software artifact: https://doi.org/10.5281/zenodo.21759702

This is an arXiv preprint and an ongoing independent research project.

(I couldn't find anything about the tag in the title in the rules, so i assumed [R] for research, if its wrong please inform)