Post-CV: When the Pig Gets Overconfident (AKA Bloated)
I Am Bloated
After finishing the Jenga stacking project from Cv1, I became bloated.
Not physically. Mentally. The dangerous kind.
The robot arm stacked a tower. The Intel RealSense D435 saw the blocks. OpenCV found
rotated boxes, center points, angles, and a “best” block. The xArm moved. Nothing
exploded. The code lives in simple-grab, if you
want to see exactly how un-magical it really is.
And just like that, my brain decided:
Ah yes. Time to do AI robotics.
That’s the bloat.
Cv1 was real progress, but it was also a narrow success. I had a working pipeline,
which is a much smaller thing than a robotics system:
- A camera image became contours.
- Contours became block candidates.
- One candidate became a grasp target.
- A human still judged whether the move was safe.
- The arm executed a mostly hand-designed behavior.
Good engineering. Also not “the robot understands assembly.” The gap between those two
sentences is the whole reason for this post.
The Pig Confuses a Demo for a Discovery
The bloating came from one specific error: I confused a working demo with a research
direction.
In Cv1 I learned enough OpenCV, numpy, RealSense depth, and robot control to make
something physical happen. It breeds a dangerous feeling: if pixels can become a grasp,
surely language can become construction, and surely a robot can become an intelligent
builder.
Then I opened the ML papers I had been hoarding, and my brain did the bad compression:
OpenCV worked + robot moved + these papers exist = I should build an AI assembly system.
The equation is thrilling and irresponsible. The honest version is smaller:
OpenCV worked + robot moved = I now own a tiny physical testbed where I can ask one
narrow question carefully.
Less inflated. Still useful. One sentence is an engineer; the other is a scientist. I want
to be the second one.
Engineer Brain vs Scientist Brain
The distinction I kept dodging:
Engineer brain asks: does it work? ship it. Success is a working artifact. The Jenga
demo passed that test on day one.
Scientist brain asks worse, slower questions:
- What exactly am I claiming?
- What is the hypothesis, and what is the null hypothesis?
- What measurement would prove me wrong?
- What am I holding constant so the result means anything?
- Is it reproducible, and what does it add that the papers don’t already say?
My Cv1 project had no claim. It had a vibe. “It stacked a tower” is a press release.
A real result reads more like “same blocks, same lighting — correction mode stood up
8 towers out of 10, one-shot managed 5, and here’s why the gap isn’t just luck.” Boring.
Specific. Checkable. The whole upgrade.
This post is me dragging the pig from engineer brain to scientist brain. The robot barely
changes. The questions change completely.
The Panic Question
What actually kept me up:
Am I doing research, or am I just badly summarizing papers written by people far smarter
than me?
If I say “I will make a robot understand language and build structures,” I don’t have a
project. I have a literature review in a costume.
But if I say:
I want to test whether short verbal corrections help a small robot arm assemble unstable
rectangular blocks more reliably than one-shot instructions.
then maybe I have something. The phrasing matters because the scope matters. A vague
ambition hides missing knowledge. A narrow question exposes exactly what must be
built, measured, and learned. I want the version that exposes me.
The Hypothesis (and Why My First One Was Cheating)
My first instinct was to write this:
H1 — letting me give short verbal corrections during a build makes the tower
succeed more often than a single up-front instruction.
It looks like a hypothesis. It even has an H1. But stare at it: of course a robot does
better when a human can step in and fix things mid-build. More help, better result. I’d be
testing whether hints help on a test, then acting shocked when they do — discovering the
obvious and calling it a finding.
The honest, interesting questions are the ones I don’t already know the answer to:
- How much does a correction actually buy me, and when does it stop working? Maybe
“move left” can rescue a wobbly second layer but is useless once the tower is four high
and already leaning. - What is the cheapest kind of help that still works? Does a dumb fixed word-list
(left,right,rotate,stop) save a failing stack just as well as a full AI that
understands “scooch it a hair toward me”? The second one is the symbols-vs-embeddings
question from the survey above — and in my setup, nobody knows the answer.
So the claim I’ll actually defend goes past “help helps” — it’s something I could genuinely
be wrong about:
A small, hard-coded correction vocabulary recovers most of the benefit of full verbal
control — enough that reaching for an LLM is, for now, theater.
If a dumb word-list keeps pace with the fancy version, that’s a real (and slightly funny)
result. If it falls apart, that’s also a result, and it tells me exactly when the AI
starts earning its rent.
One honest constraint: right now, the user is just me
This is a solo plan. I’m the one giving the commands, so I’m not collecting other
people’s opinions — there’s no “users rated it 4 out of 5 stars” here. Keeps it simple.
Keeps me out of the human-subjects rules that kick in the moment you study people instead
of a machine. (If I ever hand the mic to other testers, that becomes a
different, paperwork-heavy project. Future-me’s problem.)
Here’s the trap: I built Mode B, and I’m rooting for it. I’m the least trustworthy judge
in the room. So the measurements can’t be feelings — they have to be numbers the camera
and the robot log on their own:
- What I change: the interaction style —
one-shotvscorrection— and the question I
actually care about: how the correction is understood, hard-coded word-list vs LLM. - What I measure (auto-logged): success (defined exactly below), completion time, how
many corrections it took, and final placement error in mm and degrees against an AprilTag
target. - What I freeze: the structures, the block set, lighting, the hardware, the scan pose,
and the AprilTag target poses — so “it looked placed” becomes “off by 4.2 mm.” - What I watch out for: me getting better at it across a session (so I shuffle which
mode runs first), fatigue, and the camera slowly drifting.
Same arm. Same blocks. But now there’s a claim I could lose, and a control that stops me
from quietly cheating in my own favor.
Where This Sits in the Literature
The bloated version of me treated these papers as fuel to build bigger. The scientist
version treats each one as a coordinate — where my tiny question lives, and which
constraint I can’t ignore.
Physical feasibility — ASAP & LegoBot
ASAP plans physically feasible assembly sequences:
it asks whether each sub-assembly is stable under gravity, whether the path is
collision-free, whether the gripper can actually hold the part, and whether a robot can
execute it. Exactly the reasoning I skipped in Cv1, where I trusted that a block which
looked placeable would survive. Gravity is not polite. The universe does not orient
itself around my contour detector.
LegoBot makes the point: simple parts
don’t mean a simple planning problem. LEGO drags in exclusion zones, collision
constraints, and statics. Jenga is arguably worse: my blocks are rectangular prisms with
friction and no snaps. It is “blocks world” where the world fights back. So I will not
pretend I’m doing general robotic assembly. I’m studying one small, annoying domain where
instability is easy to observe — and that is a feature.
What I’m stealing: a constraint checker that grades a plan before the arm moves,
instead of discovering infeasibility with my face.
Shared-autonomy correction — “No, to the Right” (LILAC)
This one hit closest. Forget “robot follows
language” — its real claim is that one instruction is rarely enough. A human corrects
during execution (“no,
move right”, “tilt down a little”), feeding a learned model that produces a low-dimensional
control space the human steers. Almost literally what happened in my demo: I watched the
arm do something questionable and stepped in before it did something worse. Maybe the
interesting part was never full autonomy. Maybe it’s the feedback loop. LILAC is the
backbone of my Mode B.
Iterative VLM correction — KitchenVLA
KitchenVLA uses a vision-language model as
an evaluator that compares intent against the robot’s actual observation, detects the
mismatch, and emits a corrective action. A plan can sound perfectly reasonable and still
be wrong for the robot’s real world. “Stack the blocks into a
tower” hides which block goes first, what orientation, whether the pose is reachable, and
what to do when the depth estimate is 8 mm off. The physical world is the tax collector.
Grounding — SayCan
SayCan: language models need grounding. A model
can describe an action, but the robot should only choose actions it can actually
afford. Language × affordance. If my command vocabulary ever grows a verb the arm can’t
physically do, that verb is theater.
Discrete assembly under constraints — Speech to Reality
Speech to Reality is the most dangerous paper for my
ego because it looks like magic: speech → 3D generative AI → a physical object. But the
magic is a distraction; the real work is the constraint pipeline that discretizes and
repairs generated geometry to fit workspace limits, inventory, overhang tolerance, and
connectivity before anything is built. If I ever let an LLM “design” a Jenga structure, I
cannot accept the output at face value — I need a grader, or I’m asking the arm to
faithfully execute a hallucination.
The control — AprilTag 2
AprilTag 2 is the least glamorous paper
and probably the most useful to me right now. Cv1 did pose estimation with vision
alone, which is fine for a demo and useless for an experiment. Fiducials give me
repeatable, ground-truth poses: known targets, a way to validate calibration, and far
less time arguing with shadows. It converts my opinions into millimeters.
The axis that makes it science — the grounding survey
The paper that reframed the whole project is the
Survey of Robotic Language Grounding. It lays out a
spectrum with two poles:
- Symbols — map language to a hand-defined formal representation. Precise,
interpretable, small learning problem, easy to make safe. - Embeddings — map language straight into high-dimensional vectors that drive
low-level policy. General when you feed it enough data, but data-hungry and opaque.
My “discrete command vocabulary vs LLM” choice is this axis. The upgrade from engineer
to scientist, in one move: I’m not grabbing a tool because it’s trendy. I’m placing my
design on a named axis and defending the spot. I start at the symbolic pole —
interpretable, safe, debuggable — and slide toward embeddings only if the symbolic
vocabulary becomes the measured bottleneck. “Why didn’t you just use an LLM?” now has a
real answer instead of a shrug.
My Question, Sized Down to Honest
In a small Jenga-block assembly setup using a UFactory xArm and a RealSense camera: how
much does letting me correct the robot mid-build actually help — and is a dumb, hard-coded
command vocabulary good enough, or do I genuinely need an LLM?
Still inspired by the papers, but it’s its own thing. The domain is mine: plain rectangular
Jenga-like blocks — wood, friction, gravity, and zero snaps to save me — on a small research
arm, with the whole project pointed at comparing interaction styles over chasing
autonomy. The contribution is a clean, reproducible measurement of whether shared-autonomy
correction earns its keep on unstable blocks. The robot itself stays dumb.
This is a lot less bloated than “AI robot plays Jenga like AlphaGo.”
Thank God.
The Experiment
Two modes, same structures, same blocks, order counterbalanced.
Mode A — One-Shot Instruction
I give one command up front: “Build a three-level tower.” The system follows the
planned sequence. If the pose estimate drifts, a block tilts, or the next placement gets
risky, the robot pushes on with the original plan unless the safety stop fires. This mode
tests the fantasy that one instruction is enough.
Mode B — Iterative Correction
I can interrupt mid-execution: “move right”, “rotate clockwise”, “place it
slower”, “stop, re-detect the block.” It doesn’t need to be a genius on day one — it
starts as a small command vocabulary mapped to predictable arm adjustments (the symbolic
pole). This mode tests whether human feedback makes the system more robust, and it’s where
the LILAC-style shared autonomy lives.
What counts as “it worked” (no vibes allowed)
If I let myself eyeball it, I’ll grade generously every single time my favorite mode is
running. So success needs a hard rule the camera can enforce:
- Success = every intended level stays standing for ≥10 seconds, confirmed by the
camera — not by me nodding approvingly. - Time = from first command to final release.
- Corrections = how many nudges I had to give (zero in one-shot, by definition).
- Placement error = final block pose vs the AprilTag target, in mm and degrees.
And I keep saying this to myself: a few good runs is not proof. If one-shot fails twice
and correction succeeds twice, blame luck. To claim anything I need dozens of trials
per mode and an honest check of whether the gap beats random noise — not “it felt better.”
The line between a result and a good mood.
The Knowledge I Actually Need (the part the bloated brain hates)
The honest curriculum — the boring pieces that are the project, with “AI” demoted from
buzzword to tool. Each one: the concept, why my project needs it, and where to learn it.
1. Linear algebra: frames, rotations, transforms
Every number in this project is a coordinate in some frame — camera, tool, base — and the
job is moving points and orientations between them. Cv1 already leaned on the
eye-in-hand transform chain and a rotation matrix; the grasp orientation is where I need
to actually understand rotations rather than copy them. In Cv1 I linked 3Blue1Brown’s
chapter on linear transformations and
matrices — this is the level-up:
quaternions, which is how robot orientation is really stored.
2. Robot kinematics & inverse kinematics
The detector hands me a target pose. The arm only understands joint angles. Inverse
kinematics is the bridge — and it’s also where “is this target even reachable?” gets
answered, which matters the instant a tower gets tall. The cleanest intuition I’ve found is
the Modern Robotics course (Lynch & Park), which shows both the analytic and the iterative
numerical solution.
3. Camera & hand-eye calibration
Cv1 back-projected pixels to 3D and chained an eye-in-hand transform that I measured by
hand and marked verified so the script would refuse to move otherwise. For an
experiment, “measured by hand” isn’t good enough — I need real
hand-eye calibration (OpenCV’scalibrateHandEye) validated against AprilTags, so my placement-error numbers mean
something. The four intrinsics fx, fy, cx, cy are the same linear camera model
Prof. Nayar walked through back inCv1; calibration is just measuring them honestly.
4. Machine learning, for real
Before I’m allowed to say “AI,” I have to understand what a model actually is: a pile of
weights doing arithmetic, fed by data someone had to collect and clean. Two of the guides I
collected are about the unglamorous foundation — how training data gets labeled, and
how a model is actually trained and evaluated — which is most of the real work and none
of the hype. Start with what a neural
network even is:
5. Learned policies & RLHF — the embedding pole, the LAST rung
This is where the project could end up if the symbolic vocabulary becomes the measured
bottleneck — learned planning, and the human-feedback machinery that made ChatGPT
behave. Worth understanding precisely so I don’t reach for it prematurely: classic
reinforcement learning (David Silver’s
course), the Sutton & Barto book, and
the clearest explainer of RLHF. This is the embedding
pole of that survey axis — powerful, data-hungry, and not where a one-arm experiment should
start.
6. Physics simulation & synthetic data
Before risking real blocks (and real collisions), I can test whether a structure is even
stable in PyBullet, and — if I ever need labeled training data —
generate it synthetically the way the Kubric
paper does, instead of photographing my desk ten thousand times. Simulation is also the
honest place to ask ASAP’s stability question before gravity asks it for me.
7. Experimental design
Last and least glamorous: a project without measurement is a demo, and a measurement
without a control is an anecdote. Counterbalancing trial order, defining success before
running, and reporting the null when it wins — the difference between “the pig built a
tower” and “here’s what I found.”
The boring pieces are the project. The AI layer only gets in once they hold weight.
How I Should Develop (a ladder with exit criteria)
Each rung has to produce evidence before I climb. If a rung is weak, no amount of
language-model glitter on the top rung will save it — it’ll just hide the weakness behind
nicer words.
- Perception baseline — detect block pose from the RealSense view and log it. Exit:
repeat the same scene N times and report pose variance. - Calibration baseline — convert image/depth into base-frame coordinates. Exit: the
arm points to AprilTag-known locations within an acceptable error. - Single-block manipulation — pick one block, place it at one fixed target pose.
Exit: repeated trials succeed without me editing code between attempts. - Three-block fixed plan — build one tiny structure from a predefined sequence.
Exit: success rate, timing, and failure modes are logged. - Correction vocabulary — add
left,right,rotate,lower,stop,re-detect. Exit: each command maps to a predictable, repeatable arm adjustment. - One-shot vs correction experiment — run the same structures in both modes, dozens of
trials each. Exit: compare the four logged metrics, with the order shuffled. - Only then, flexible language or learned planning — slide toward the embedding pole
only if the fixed vocabulary is the measured bottleneck.
Current Conclusion (It’s Still Just a Plan)
The new mission:
Build a small Jenga-stacking setup where my verbal corrections are part of the control
loop — then measure how much they actually help, and whether a dumb word-list does the job
an LLM was going to take credit for.
Right now it’s a plan, and the honest risk is the boring one: I spend six months
perfecting calibration and never run trial number one — a gorgeous robot with zero data,
which loses to a duct-taped rig that actually measured something. So the rule I’m giving
myself is simple: get to the boring part — the part where I run it forty times and write
down what happened — as fast as I possibly can.
The robot isn’t any smarter than it was in Cv1. I just stopped letting it lie to me.
At least now the pig is only moderately bloated.