Thread of 2 posts
Oh, someone already sent this (sorry; your instance was failing to load so I didn't read it until I checked again and it did work). Oh well. I don't think it's very hard to write your own implementation of something like this - the LLM outputs a probability distribution over the next token, so you just need to repeatedly run forward passes through it and do something something arithmetic coding.