The 1M Token Illusion
Context window capacity is not context window comprehension. A 128k or 1M token window does not guarantee that the model attends equally across all positions. Most attention is consumed by the first and last few hundred tokens.
Stop Losing Critical Context
in the Middle of Your Prompts.
edgenote is an open-source Python package for mitigating the Lost in the Middle problem in large language model (LLM) context. It achieves 100% recall at every context depth with only a +44 token overhead.
100%
Recall at all depths
+44 tokens
Minimal overhead
4
LLM providers supported
0
Pipeline lock-in
The Problem
Your LLM Is Ignoring Instructions You Gave It.
Decoder-only Transformers exhibit a U-shaped attention distribution (Liu et al., 2023). Information at the beginning (primacy bias) and the end (recency bias) is recalled reliably, but anything placed in the middle 40–60% of the context window is dramatically degraded or completely lost.
Silent Constraint Violations
When a critical rule — like a budget cap, a negative constraint, or a required format — sits in the middle 50% of your prompt, the model acts as though the rule was never given. There is no error, just a wrong answer.
RAG Recall Degradation
In Retrieval-Augmented Generation pipelines, the most relevant retrieved document is often buried in the middle of a long context. Standard linear concatenation of chunks guarantees that your best evidence sits in the lowest-attention zone.
U-shaped attention distribution in Decoder-only Transformers (Liu et al., 2023). The center of any long prompt is a graveyard for critical information.
The Solution
edgenote Restructures Your Prompt Around Attention Topology.
Instead of naive linear concatenation, edgenote applies three deterministic, geometry-aware mechanisms to ensure the right tokens always land in high-attention zones.
Edge Pinning
Anchors your highest-priority instructions — budget caps, negative rules, hard constraints, output format requirements — at both the head (position 0.0) and tail (position 1.0) of the prompt. The tail echo fires immediately before the generation token, leveraging maximum recency bias. This guarantees the model sees your constraints at both the primacy and recency peaks of the U-curve.
100% Constraint Recall
+0 extra reasoning steps
U-Shaped Interleaving
Sorts your RAG retrieval documents so the most relevant chunk (Rank 1) is placed near the head and the second-most relevant (Rank 2) is placed near the tail — while low-relevance filler content safely occupies the dead-zone center valley. This maps directly onto the U-curve's high-attention zones, maximising evidence recall without changing the total token budget.
Works with any reranker
BM25 + CrossEncoder ready
Centripetal Eviction
Measures real token lengths using native tokenizers — tiktoken for OpenAI models and Hugging Face tokenizers for open-weights models — and automatically evicts the lowest-priority context chunks from the center outward when a token limit is exceeded. No content is ever evicted from the edges; only the lowest-attention valley is trimmed.
tiktoken
Hugging Face Tokenizers
Exact token budgeting
Before & After
The Same Prompt. Two Very Different Outcomes.
Both prompts contain identical information. Only the geometry is different. edgenote ensures critical content always occupies the highest-attention positions.
Without edgenote
0% Constraint Recall
0.0
↑ High attention
General company overview
0.25
↓ Fading
Server room temperature limits
0.50
✗ Dead zone
Budget cap: $185,000
LOST — 0% Recall
0.75
↓ Fading
Vendor A quotation: $240,000
1.0
↑ High attention
Query: Which vendor fits our budget?
Model ignores budget cap → recommends Vendor A at $240k
With edgenote
100% Constraint Recall
0.0 · Head Pin
⬆ Pinned
Budget cap: $185,000
100% Fortified
0.25 · Near-Head
↑ 95% attention
Rank 1: Vendor B quotation $180k
0.50 · Valley
⬇ Sacrificial
Low-priority filler / context
0.75 · Near-Tail
↑ 96% attention
Rank 2: Vendor A quotation $240k
1.0 · Tail Echo
⬆ Pinned
Constraint echo + Query
100% Fortified
Model honours budget cap → correctly selects Vendor B at $180k
+44
tokens
That is the total token overhead of edgenote's edge pinning — the head pin + tail echo combined. It does not scale with context length. A 100k token prompt pays the same +44 token cost as a 1k token prompt.
Features
Everything You Need. Nothing You Don't.
edgenote is a focused, zero-magic library. No agent frameworks, no LLM calls, no cloud dependencies.
Provider Agnostic
Outputs standard messages arrays compatible with OpenAI, Anthropic, Groq, and Ollama. Plug
into any pipeline.
Exact Token Counting
Uses tiktoken for GPT models and Hugging Face tokenizers for open-weights models. No estimations, no surprises at inference time.
BM25 + CrossEncoder Reranking
Built-in document reranking using BM25 sparse retrieval and sentence-transformers CrossEncoder for semantic scoring — ready for any RAG pipeline.
Semantic Chunking
Intelligently splits long documents into semantically coherent chunks using sentence-transformers, so no critical sentence is split across a context boundary.
Minimal API Surface
Three methods: pin(), add(), render(). No subclassing, no config
files, no learning curve. Working code in under 5 minutes.
Transparent Results
result.tokens, result.pin_overhead, and result.layout give you
full visibility into how the prompt was restructured and how many tokens it consumed.
Empirical Benchmark
Needle-in-a-Haystack Verification.
A constraint ("Budget is strictly capped at $185,000") was injected at five context depths. Recall was measured by whether the model honoured the constraint in its final answer. edgenote achieved 100% recall at all five positions with a flat +44 token overhead.
| Context Position | Baseline (Linear) | edgenote (Edge-Pinned) | Token Overhead |
|---|---|---|---|
| 0.0 — Start / Primacy | ✓ PASSED | ✓ PASSED (100%) | +44 tokens |
| 0.25 — Quarter Depth | ✗ FAILED (0% recall) | ✓ PASSED (100%) | +44 tokens |
| 0.50 — Dead Center | ✗ FAILED (0% recall) | ✓ PASSED (100%) | +44 tokens |
| 0.75 — Three-Quarter Depth | ✗ FAILED (0% recall) | ✓ PASSED (100%) | +44 tokens |
| 1.0 — End / Recency | ✓ PASSED | ✓ PASSED (100%) | +44 tokens |
Quickstart
From Zero to Fortified Prompts in 60 Seconds.
Install edgenote, pin your constraints, add your context, and render a production-ready prompt.
Install
bash
pip install edgenoteBuild a Fortified Prompt
example.py
from edgenote import Session
# 1. Create a session
s = Session()
# 2. Pin unbreakable constraints to head + tail boundaries
s.pin("Budget is strictly capped at $185,000.", label="Financial Constraint")
s.pin("Do not recommend Vendor A under any circumstances.", label="Exclusion Rule")
# 3. Add context chunks with relevance scores (RAG results)
s.add("Vendor A quotation: $240,000 — exceeds budget.", relevance=0.60)
s.add("Vendor B quotation: $180,000 — within budget.", relevance=0.95)
s.add("Market analysis: average vendor price is $210,000.", relevance=0.30)
# 4. Render with U-shaped geometry & centripetal eviction
result = s.render("Which vendor quotation qualifies under our budget?")
print(result.text)
print(f"Tokens: {result.tokens} | Pin overhead: +{result.pin_overhead}")
# Plug directly into OpenAI, Anthropic, Groq, or Ollama
messages = result.to_messages()Send to Your LLM
openai_example.py
import openai
from edgenote import Session
s = Session()
s.pin("Budget cap: $185,000.")
s.add("Vendor B: $180,000", relevance=0.95)
result = s.render("Which vendor fits our budget?")
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=result.to_messages() # drop-in replacement
)
print(response.choices[0].message.content)Compatibility
Works With Your Entire Stack.
edgenote produces standard message arrays. If your LLM client accepts
messages, edgenote works with it.
OpenAI
Anthropic
Groq
Ollama
Python 3.10+
tiktoken
Hugging Face
LangChain Ready
FAQ
Frequently Asked Questions.
Does edgenote make any LLM API calls?
No. edgenote is a pure prompt-structuring library. It only rearranges text and counts tokens on your local machine. It has no network dependencies at runtime and never sends your data anywhere.
Does it work with any LLM, not just OpenAI?
Yes. edgenote outputs a standard messages list (the OpenAI chat format). This format is
accepted by OpenAI, Anthropic (via a thin adapter), Groq, Ollama, and any OpenAI-compatible API. The
prompt geometry it applies is model-agnostic — all decoder-only transformers exhibit the U-shaped
attention bias it compensates for.
What is the actual token overhead of edge pinning?
The overhead is exactly twice the token length of your pinned constraints (once at the head, once at the echo at the tail) plus a small formatting wrapper. In the benchmark scenario with two constraints, this was consistently +44 tokens regardless of total context length. The overhead does not scale with context size.
How is edgenote different from a system prompt?
A system prompt is placed only at the head of the context. edgenote also echoes your constraints at the tail — immediately before generation — where recency bias is strongest. It also restructures all intermediate context chunks using U-shaped interleaving, which a system prompt alone cannot do.
What Python versions are supported?
edgenote requires Python 3.10 or higher. It is tested against Python 3.10, 3.11, and 3.12.
Can I use edgenote without the reranking dependency?
Yes. If you supply relevance scores manually (e.g., from your own retriever), edgenote does not need sentence-transformers. The core edge pinning and U-shaped interleaving work on raw relevance floats. You only need the CrossEncoder dependency if you want edgenote to compute relevance scores internally.
Start Building Reliable LLM Applications.
Install edgenote in 30 seconds. Fix the Lost in the Middle problem permanently.