Model & weights: [BrunoHays/embeddinggemma-2-nullspace-audio](https://huggingface.co/BrunoHays/embeddinggemma-2-nullspace-audio)

I wanted to share an experiment on `google/embeddinggemma-2` (577M) for MAEB, the 30-task audio benchmark on the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard). By projecting intermediate Conformer layers into the unused null space of the model's final projection matrix using only PCA, the model's macro score jumps from **49.3% to 54.4% (+5.1 points)**, moving from **Rank #26 to #1** on `MAEB audio-only` (and **#16 to #1** on full `MAEB`), ahead of 8B-9B multimodal models.

### 1. Last-layer filtering in text-aligned audio encoders

Across the MAEB leaderboard, LLM-backed Omni embedders score 95-99% on multilingual text-to-audio retrieval, but drop to **1%-6% on unsupervised audio clustering** and ~20% on vocal emotion (`CREMA-D`). Aligning an audio encoder to a text LLM filters the final layers: lexical content (*what was said*) is kept, while paralinguistic and acoustic traits (*timbre, pitch, emotion*) are discarded.

### 2. Probing the 12 Conformer layers

Layer-wise speech studies ([Pasad et al.](https://arxiv.org/abs/2107.04734); [Bouziane, Mdhaffar & Estève](https://arxiv.org/abs/2610.21941)) show that speaker, prosodic, and semantic attributes peak at different depths inside speech encoders. Probing all 12 Conformer layers of `embeddinggemma-2` confirmed this:

- **Layer 2 (mean + std pooling):** `VoxPopuli` gender clustering goes from **0.05% (final output) to 66.6%** (Layer 3) / **65%+** (Layer 2), and `IEMOCAP` gender classification goes from **66.4% to 83.0%**.

- **Layer 6 (mean pooling):** `CREMA-D` emotion classification doubles from **19.7% to 39.5%**.

### 3. Why EmbeddingGemma-2 has a 256d null space

In `embeddinggemma-2`, the text backbone hidden size is **512**, and the terminal projection matrix `W` has shape `768 x 512` with no activation after it.

- Because `W` maps linearly from 512d to 768d, every text vector lies in a 512d subspace.

- Taking the SVD of `W` gives the remaining **256 orthogonal directions (the null space)** that text vectors never use.

- Any audio vector injected into those 256 directions has zero dot product with all text vectors.

### 4. Zero-training PCA injection

I compressed Layer 2 and Layer 6 down to 256 dimensions using PCA (fitted on 1,400 clips from 4 external audio datasets, with no MAEB/MSEB data) and injected them into the 256d null space before normalizing:

- **Text embeddings:** bit-for-bit unchanged.

- **Text-to-audio retrieval (100+ languages on FLEURS, CommonVoice, Google SVQ):** unchanged within 0.05 points.

- **Audio-to-audio tasks:** gender clustering jumps **0.05% -> 63.1%**, artist retrieval **55.2% -> 93.8%**, emotion **19.7% -> 40.2%**, and environmental sounds **38.8% -> 56.3%**.

### 5. Caveats

- **No Matryoshka on audio:** Because there is no retraining, we need all 768 dimensions for orthogonality to hold and lose the 512/256/128 truncation options.

- **Spoken-intent trade-off:** Adding fixed acoustic features slightly scales down the 512d semantic component after normalization, which hurts audio-to-audio tasks where speaker voice should be ignored (spoken topic classification `SIBFLEURS` drops `57.2% -> 21.3%`, spoken intent `MInDS1

Still, this shows how much untapped signal can be extracted from overtrained foundation models even with zero training. With actual training restricted to `U_perp`, I'm certain that we could go much further.

I've been thinking about AI agent safety, and I built a test harness to check my own results. I think it may be useful to people who study models too.

It records where each tool argument came from before the tool executes, measures whether the same input gives the same decision, and tests whether a model takes arguments from someone else's text instead of its user's.

If you modify the harness, it can probably be used for much more.

https://github.com/Jang-woo-AnnaSoft/execution-state-preflight/blob/main/hooks-example/README.md

Hi, just [sharing a project](https://blog.christianperone.com/2026/10/pandora-m1-a-training-monitor-box-64-nodes-x-8-gpus-support/) I started a few months ago in case anyone is interested. The main idea is that you will have a desktop front-panel where you can monitor distributed training across multiple nodes.

Hi everyone,

I wanted to share an open-source repo collecting 20 exploratory studies focused on Next-Concept Modeling (NCM), mechanistic interpretability (like IOI in GPT-2), and model controls.

Link: https://github.com/Quantix-eng/ncm-experiment-notes

This is an independent project documenting empirical experiments on how transformers process and edit internal features. No performance breakthroughs here—just honest documentation of observations alongside rigorous controls.

The repo covers NCM on structured tasks (like Sudoku), mechanistic interpretability in GPT-2, and toy parallel models. I've included all source code, saved results, and interpretations for every run.

For full transparency, I’ve categorized every outcome, including the mixed and completely negative results. AI tools were also heavily used during the planning and coding phases.

Would love to get your feedback on the setups or interpretation limits.

TL;DR: AI written reviews, and AI-writing in papers

I was going to give this a title "An Obituary of the Review Process", but in the interest of not sounding too defeatist, I will just say "Critical Condition" because of AI writing, both on the author side and the reviewer side.

I am a reviewer for AAAI 2027 and ICLR 2027. Since Phase 1 and Phase 2 was completed, I can now see other reviews. I can bet good money that about half of the reviews in the papers I'm seeing are AI-generated completely, or edited after AI-generation. This is subjective opinion and I cannot prove it but I will try to convince you - at least convince you that it's a problem that needs much much more attention than is being currently given.

I have been reviewing for top conferences (NeurIPS, ICML, AAAI, ICLR) since 2022, in total about 10 times. I have not felt the same way about reviews, or even about papers, ever, except until recently.

Author side: There was only one paper in the 3 ICLR submissions that felt 100% human-written (now if that isn't either - that'll make me very sad). The others had some elements that smell like AI but you can't prove it e.g. excessive use of very descriptive headings with colourful, marketing-like language e.g. a heading called "The theory predicts performance, and empirical evaluations confirm it", or very dense long sentences where you lose track of the thought where instead of already used terms their synonyms are used and then later even more synonyms, and so on.

Reviewer Side, talking about the reviews I suspect: There is very little variety in the language, between the reviews, and within the same review. One review puts a field "questions for authors" in the weaknesses section, which I suspect is because the field name is changed this time around to "specific points for rebuttal". They've put "N/A" in the new field. The paragraph lengths are almost exactly the same. As if an LLM was told to give 5 strengths, 5 weaknesses, 5 questions. There are also signs of extra effort that a human rarely does e.g. instead of just saying "provide peer-reviewed references instead of arXiv", it also adds the arXiv number; or another review which not only puts doi link (this much is fine), but then in an extra piece just writes "(doi.org)" which seems like an attempt by an LLM to give a source. Reviewers are usually pressed for time, they usually try to communicate so the authors understand, not that everything is exact. These reviews are written as if for public consumption. Then there are smart quotes - which one can think are there because maybe Word or something was used to write them; but seen in light of everything else, now these quotes also smell like AI to me. (I used to not see smart quotes this often in earlier years. Word made quotes smart back then as well.) Unicode symbols instead of LaTeX for things such as \tilde{d}, dashes that are not hyphens - in one review even for a minus sign, and so on.

There are at least two AAAI submissions where there are only two reviews and I strongly suspect the other review to be AI-written.

Everything is broken. I am sorry I have no concrete ideas how to fix this, so far. Do you?

Except one maybe, disallow AI usage even for polishing styling and tone? Just completely disallow AI writing. The author feels it's a change in tone and style only but the reader can sometimes very quickly _feel_ it. Before these powerful LLMs we were not judging papers based on language anyway. I know it's an honor system, but I think that's okay. Saying "you can use AI to polish" perhaps also creates an expectation to polish using AI.

I’ve uploaded a minimal, self-contained implementation of manual autograd for a transformer-based classifier in PyTorch. It can help build intuition for what autograd is doing under the hood and is a useful hands-on reference for low-level differentiation in Transformer models, such as writing custom backward passes and tracing how gradients flow through attention blocks.

🐙 GitHub:

https://github.com/ifiaposto/transformer_custom_autograd/tree/main

📓 Colab:

https://colab.research.google.com/drive/1Lt7JDYG44p7YHJ76eRH_8QFOPkkoIwhn

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).

Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:

-> robust SDFT implementation that runs under ZeRO-3

-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes

-> Saving and resuming EMA teacher checkpoint

-> working deepspeed configs that fix memory spikes when GPU memory is tight

I tested on Qwen3.5 and ToolAlpaca.

Repo: https://github.com/ifiaposto/sdft_zero3

Feedback/ more examples/ issues welcome!

It's a 50 hour course for beginners in AI/ML (obviously it's fully free). Starts out with linear algebra, derivatives, statistics and then linear regression, neural networks and backpropagation etc...

kinoml.tech

Hey! We published a set of 6-step LoRAs, one without the upsampling step: https://huggingface.co/PrunaAI/P-Q-Image-2.1

- Generate or edit images in just 6 steps instead of 40. Use 6-steps for preserving inputs when editing and 6-steps-upsampled for improved visual quality and shorter end-to-end generation.
- 6-step distillation provides improved quality with few generation steps. 6-steps-upsampled is the first distillation that removes the need for a separate prompt-upsampling step. This means 41 to 80 seconds saved, depending on your prompt style.
- The LoRA adapters load directly on top of Qwen-Image-2.1, so the pipeline stays unchanged. Few-step distillation cuts the diffusion cost, and the upsampled adapter also removes the prompt-upsampling step before image generation.

Chain-of-thought transformed AI reasoning, but it is limited to words.

What's the physical-world analogue?

Introducing Ψ-0.6, one model that reasons in *streams of thought*: words, depth, motion, camera pose, and imagined scenes. And it learns when to use which!

• Ψ-0.6 surpasses leading open-weight models on complex spatial reasoning (MindCube, zero-shot).
• On FlowBench, it outperforms other world models at predicting how objects move. Trained on simulated rooms, it transfers to real robot tabletops.
• One model as perceiver, VLM, world model, reasoner, and policy. RL learns which representation to reason through.

Thread (with video): https://x.com/khai_loong_aw/status/2115099953433600011

• Paper: https://arxiv.org/abs/2610.32177
• Website: https://neuroailab.github.io/psi-website/blog/psi-0-6.html
• Code: https://github.com/neuroailab/psi-0.6

Hi guys,

Self-distillation methods, see Self-Distillation Fine Tuning SDFT, usually use an EMA (Exponential Moving Average) teacher to get synced and stable targets. I wanted to ask the community for research papers/ works that consider different types of teachers that also hold these properties (they eventually also improve along with student) but are stable (prevent student from drifting).

any links to repos/ papers would be appreciated.

thanks!

For my master’s thesis I built an accelerator for diffusion-based action generation in robotics policies. Offline, it looks great: the median absolute deviation between the accelerated output and the original model’s output is extremely small, so I concluded it reproduces the model’s behavior faithfully.

When I deployed it on a real robot for a task (picking up a book and placing it on a shelf), it did not work as well as the original model. The offline numbers did not predict the real-world result.

I want to submit this to MLSys. My offline results are strong, but the deployment result is negative. My questions:

1. Is it acceptable to submit with only the offline results, or should I include the failed deployment?
2. If I include it, how do I frame it so it adds to the paper instead of sinking it?
3. Why might a tiny median deviation still lead to task failure? My guesses are tail errors and error compounding in closed loop, but I would like other views.
4. What additional metrics would you report (percentiles, per-step error, closed-loop success rate in simulation)?

Any experience from people who have submitted systems papers with negative real-world results is very welcome.

A 3×8 array of clock faces is one of the examples in the SenseNova 6.8 Code2Video showcase. Other entries focus on letterforms, including A to Z and FRAME / FORM / FLOW.

The scientific scenes include Earth and spacecraft clocks, molecular diagrams, and a ball-and-spring setup with separate energy displays. These bring repeated objects, labels and spatial relationships into the same collection as the character-animation examples.

https://github.com/OpenSenseNova/SenseNova-6.8-Code2Video

Hey, I've been working on a PyTorch optimizer called ApexW and figured I'd share it here to get some feedback from people who know more about this stuff than I do.

The main goal is to reduce optimizer-state memory usage and see how much faster I can make optimizer steps on larger GPU workloads. I've been experimenting with compact optimizer states and a Triton-based execution path.

In my current tests, the Triton + compact-state path was around 1.5x faster than PyTorch's fused AdamW on workloads with a few large tensors, and the optimizer-state memory usage was roughly cut in half.

The results aren't all good, though. Those numbers are from optimizer-only benchmarks, not full model training. The automatic execution mode doesn't consistently get the same speedup yet, and workloads with lots of small tensors are much slower than PyTorch's fused optimizer. So I'm definitely not claiming this is a better AdamW replacement for every workload.

I want to test it on more realistic models and different GPUs, and I'd appreciate a second pair of eyes on both the implementation and the benchmarks. the Reality Check (The Trade-Offs)

I want to be completely realistic here—this is an engineering trade-off, not magic:

1. Python Loop Overhead on Small Tensors: When testing a layout with many tiny tensors (many-small test), ApexW’s Triton kernels suffer heavily from Python's launch latency. For example, at 128M parameters spread across many small tensors, PyTorch native was 16.1 ms/step while ApexW compact was 76.0 ms/step (~0.21x speedup). If your model is composed of many tiny layers, standard PyTorch is significantly better.
2. End-to-End Impact: While the optimizer step itself is up to 1.5x faster, this step only represents a fraction of a full training step (which is dominated by forward and backward passes). In my end-to-end MLP test, PyTorch still runs slightly faster overall because of C++ deep integration.

Why/When should you care?

If you are training larger deep learning models (like Transformers or LLM fine-tuning) where you are constantly hitting Out-Of-Memory (OOM) barriers,**,**cutting your optimizer state memory strictly in half lets you increase your batch size or scale up your model size without upgrading your hardware.

GitHub: https://github.com/omen42515-prog/apexw/tree/main

If you've worked on PyTorch optimizers, Triton kernels, or training larger models, I'd be interested to hear what you think. Especially curious if there's an obvious issue with my benchmark setup or an important test I'm missing.

Papers with Code is now open-source with an Apache 2.0-license.

The goal is to collaboratively create the best paper interface with the community!

Over the past couple of months, I've been working with my buddies Claude and Codex on creating a website that tracks trending AI research, benchmarks and state-of-the-art (SOTA). It's a revival of the original Papers with Code website, which Meta acquired and later shut down.

The website is pretty straightforward:
- Python FastAPI backend
- React frontend

The app itself is hosted on a Hetzner Virtual Private Server (VPS). The code is also hosted there as a single monorepo.

The frontend mostly calls various API endpoints to visualize papers. The API itself is also publicly available, as is an accompanying MCP server.

I've been getting cool feature requests lately, so rather than just working on it myself, I thought it would make a lot more sense to share the codebase openly so anyone can contribute.

Code: https://github.com/huggingface/paperswithcode

I'm working through a free series on GPU kernel engineering (my own channel; I posted the roadmap here earlier: https://www.reddit.com/r/MachineLearning/comments/1xa4h2g/open_gpu_kernel_engineering_series_naive_gemm_to/ ) . Session 1 is a setup session. While preparing it I collected a few things that cost me time and that I rarely see in one place, so I'm sharing them here directly. The video and code are linked at the end.

1. Check compute capability, not just "has an NVIDIA GPU"

nvidia-smi --query-gpu=name,compute_cap --format=csv

7.5 is Turing, 8.0/8.6 Ampere, 8.9 Ada, 9.0 Hopper. This decides what you can do:

- CUDA 13 dropped Maxwell, Pascal and Volta. For those cards, install a CUDA 12 toolkit.

- GTX 16-series is Turing but has no tensor cores.

- Tensor core work needs RTX 20-series or newer (or a T4); Ampere-specific instructions need Ampere or newer.

- FP8 needs Ada or Hopper. Hopper-specific features (TMA, WGMMA) need an H100, which most people will have to rent.

1. Driver, toolkit and PyTorch's runtime are three different things

The CUDA version in the nvidia-smi header is the newest version your driver supports, not what's installed. pip-installed PyTorch bundles its own CUDA runtime, so torch.cuda.is_available() returning True says nothing about whether nvcc exists. On WSL 2, don't install a Linux driver inside the distro; use the WSL-Ubuntu toolkit-only installer, because the Windows driver is used.

1. GPU launches are asynchronous, so naive timing measures queueing

A Python timer around x @ y mostly times the launch, not the work. Warm up, synchronize, use CUDA events, average over several runs:

def bench(fn, iters=20):

for _ in range(3):

fn()

torch.cuda.synchronize()

s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)

s.record()

for _ in range(iters):

fn()

e.record()

torch.cuda.synchronize()

return s.elapsed_time(e) / iters # ms

For a matmul, throughput is about 2*M*N*K FLOPs divided by time. Record the dtype next to the number, since FP32 and BF16 baselines differ a lot.

1. Profiler counters: ERR_NVGPUCTRPERM

By default, non-admin users can't read GPU performance counters, so Nsight Compute fails with this error. Fixes: run ncu with sudo (use the full path, since sudo usually drops your PATH), or have an admin enable counter access for all users. On Windows/WSL it's enabled from the Windows side. If you rent GPUs, test this before committing to a provider, because managed container platforms can block counter access.

1. Smoke test before anything else

A tiny add-one kernel with an error check on every CUDA call, built with nvcc -arch=native, then nsys and ncu run on it once. If that works, the driver, toolkit, GPU and both profilers are all fine.

The session also covers the prerequisites (pointers, flat matrix indexing, matmul and its FLOP count, PyTorch basics) and compares ways to get a GPU. It runs about 40 to 45 minutes.

Video: https://youtu.be/h_zWngMWHBE

Code: https://github.com/prasoonmhwr/gpu_kernel_engineering (folder: session-01-prerequisites-setup)

Caveats: the commands are written for Ubuntu and I haven't tested every distro.

Questions for people who do this daily:

- What setup trap cost you the most time? I'm sure I've missed some (Secure Boot and driver loading, container toolkits, mismatched host compilers).

- Is anything above wrong or out of date for your setup?

Recently I've been getting bombarded with reach outs from companies that build RL environments and sell to frontier labs like Anthropic, OAI, etc. I'm not really understanding how and why these companies exist.

Lets say you somehow create an environment that no other model is able to perform well on. Your second criteria would be that this environment is suitable for RLVR, perhaps you have some metric of gradient information per example in the dataset. So you have to prove to the lab you are selling this on, that 1) their model cannot do this task currently and 2) this task is solvable via RL.

At that point, why couldn't you just approach a VC (most of these companies are startups with a good amount of funding anyways), show them scaling laws + your current RL env and just raise money to train your own model? We already see open source models tend to lag behind frontier ones by ~3 months or so, it sort of implies the frontier is mostly new RL environments and stuff (or distillation methods are really good). I'm kinda operating under the impression that pre-training/architecture/infra for large models is largely much easier nowadays, but maybe this is just not the case?

I guess an answer to my question would be either 1) I am totally wrong regarding my assumptions on the difficulty of pre-training + infra (which from my experience in industry does seem to be much easier) or 2) actually collecting all of these other, past, RL environments is still hard and only certain labs have access to all of them or 3) all of these companies are or eventually will train their own models?

I am a 3rd year PhD student in a small lab working on ML in healthcare. My PI is new (we have no PhD graduates yet - lab is 4 years old).

I want to publish in a top ML conference (I know its necessary for an ML researcher job).

And my PI is saying you need to do method research and I don't know how to do it?

We don't have the amount of data to do foundational models. Our data is very small.

I feel like my PI had me jumping around topics he was excited about for the 1st two years of my PhD and nothing paned out or I had no idea what I was doing.

I started thinking of a certain idea but I needed guidance on how to implement it... My PI was no help and tried pushing me to a new idea. He is usually very "attacky" to my ideas, I always get the feeling I need to prepare to really really defend any idea I have to him.

It's either too incremental, too applied, or not impactfull enough. I tried asking what exactly is a method research? His example was the transformers paper, and his own paper (which was incremental according to his current criteria).

I now have no idea what I am doing, where I am going or what I am going to do...

I still have zero idea what my PhD research question is...

Do you guys have any advice?

* this is my 1st time posting so please be gentle 😅

For the past year I have been studying using a fast K-dim hermite spline manifold algorithm, called K-Splanifolds or KS, for general purpose fast regression. In this video I show the third version, KS3, compared to MLP and older KS versions fitting a 4D mandelbulb via gradient descent on random samples. KS3 converges faster and has nearly identical latency on a GPU as MLP.

KS blocks are suitable for a wide range of regression problems and I have found KS3 ranges from considerably faster convergence as MLP (noise removal, geometric fit, financial prediction) to neck and neck (LLMs). You can view code for all three versions of the algorithm and a paper for KS1 here: https://github.com/curvedinf/k-splanifolds

I am an independent and if anyone would like to explore this with me, let me know. Also if anyone would like to donate access to 8x H100, I would like to attempt a world record nanogpt speedrun with KS3 using configurations I have found beat the WR on proxy hardware.

This is a "survey of current practices" type of research for a niche area, nothing earth-shattering. I’ve written a 2-part Medium series (free, no paywall) on scoring Intelligent Document Processing (IDP). It focuses on Key Information Extraction (KIE) from structured documents, especially forms. I lean heavy on metrics derived from the confusion matrix: different ways to map items such as blanks or misreads to the four possible values, how to count them, when to use each metric.

• Scoring IDP, Part 1: Confidence Is Not Correctness
• Scoring IDP, Part 2: Approaches to Evaluating Form Extraction

Topics I include, but not limited to: recording raw outcomes before mapping them to the confusion matrix (correct, misread, missed, invented, correct blank); confusion matrix scoring for text fields and checkboxes; misread conventions and their effect on scores (FN-only, FP-only; FN & FP); how blanks, NULLs, unreported checkboxes, and unreadable values are counted; F1 vs. Recall vs. Accuracy, including how correctly identified blanks inflate Accuracy; Jaccard Index (IoU) as Accuracy without True Negatives, and its conversion to F1; Precision, Specificity, and False Positive Rate for LLM hallucination control; micro, macro, and weighted F1; choosing a headline score and stating the convention behind it; confidence vs. correctness and calibration (ECE, MCE); and Key Information Extraction (KIE) vs. OCR metrics (CER, WER).

I believe these would fall under "machine learning scoring techniques" applied to Intelligent Document Processing (IDP)--how to transparently and fairly measure (score) rather than how to implement. It may also help folks understand how to read results coming from different benchmarks.

I try not to recommend one opinionated way of doing things and instead cover the variations I’ve run into and the tradeoffs between them. I’m not endorsing any vendor; the series is about scoring itself.

I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).

Results

• vs a grid (equal compute, 3 seeds): the best fixed cell reached 95% val acc in 317 ± 29 steps; PBT took 2667 ± 2244. PBT was slower in every seed, but the grid’s corner already contained the answer.
• Dead-zone start (lr 1e-4 to 1e-3, wd 0.01 to 0.08, where no fixed setting groks): fixed 0/3, slow PBT 0/3, fast PBT (every 100 steps, ×0.5/×2) 3/3 at about 6850 steps.
• Its path: lr drops to the floor, then wd climbs to the ceiling, then lr rises. Val loss rewards low lr × wd early on, which is the wrong direction for grokking, then flips sign later.
• Ablations: a random donor still escapes, just later. With no selection, members grok and then the population loses it.
• 10 seeds: wide random search hits 95% at a median of 300 steps, vs 6650 for PBT from the dead zone. PBT also sometimes collapses into the lr-floor/wd-ceiling corner and never leaves.
• Replay: the winning lr/wd schedule with no PBT, on fresh inits: 55/60 reach 95%. Constant knobs at the schedule’s geometric mean: 0/60. The schedule carries the result, not the weight copying.

Takeaway: PBT is a rescue tool for a bad range, not a speed-up.

Caveats: one task, one model, one split. 3 to 10 seeds per test, with p-values mostly above 0.05. The replayed schedules are post-hoc winners. I haven’t tested a range with no grokking region at all, which is where PBT should beat random search.

Write-up with interactive figures: https://shauryasharma.tech/pbt-grokking.html
Code and all results: https://github.com/Shaurya-34/grok-pbt

everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess

The tasks

1. Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).
2. Rename a function across 3 files with messy spacing. All 5 passed.
3. Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".
4. Count the errors in 3 log files. All 5 got 43.
5. Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.

Setup

One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw

Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.

I'm interested in how people build datasets for fine-tuning open-weight models, especially when it is about making a model handling edge cases than just perform well on common examples.

I'm curious about the data side of the process.

• Finding edge cases: How do you identify edge cases that are rare or unusual. Also how do you make edge cases on an existing dataset to make it more specialized for what you need.
• Generating examples: Do you write them by yourself, or use a good model to generate synthetic data, mine real data, etc?
• Quality control: How do you catch incorrect, repetitive, or unrealistic examples?
• Coverage: How do you know when your dataset covers enough of the failure cases that matter?
• Measuring impact: Have you had cases where adding more edge cases actually made the model worse or failed to improve it?

Nanbeige released Nanbeige4.3-33B as the teacher for its planned smaller agents, including an upcoming 3B model. the weights are Apache 2.0, and the model card lists a 512K context window.

the team reports 80.9 on Terminal-Bench 2.1 versus 73.0 for Qwen3.8-27B, using the same harness and inference settings within each benchmark. it does not win everywhere. on SWE-Bench Pro, the reported scores are 60.3 versus 61.7. these are the releasing team's evaluations, not independent replication.

one practical detail for reproducing the agent results: the evaluations retain earlier reasoning with preserve_thinking=true. the card recommends that setting for multi-turn tool use, but not general chat. comparing task success and total generated tokens with that setting held fixed would be more useful than treating the headline scores as a deployment ranking.

source: https://huggingface.co/Nanbeige/Nanbeige4.3-33B

Thank you for your efforts in protecting the community from AI slop.