_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
URI Visit Hacker News on the Web
COMMENT PAGE FOR:
URI Paper2Code: Automating Code Generation from Scientific Papers
wzdd wrote 11 hours 26 min ago:
I did this recently with a forward-mode AD paper, by just pasting the
PDF into Claude. Like everyone, I've had mixed results with Claude
coding, so I wouldn't bet my life on the output, but Claude was able to
produce something for Pytorch that worked first go, had appropriate
performance characteristics, and it was able to convincingly explain
the connection between parts of the generated code and the paper. I was
impressed.
brundolf wrote 14 hours 26 min ago:
Not what OP is about, but idea I just had:
We should have the technology now to hand-write pseudocode on a piece
of paper (or whiteboard or chalkboard), and have it translated and
executed. Maybe you even hook up a projector, and project the output
back onto the board
ks2048 wrote 17 hours 33 min ago:
So who has a code2paper model that we can hook up in a loop?
somethingsome wrote 20 hours 48 min ago:
I like the idea of having automatic code creation from papers, but
Iâm scared of it.
Suppose you get a paper, you automatically implement the code, and then
modify it a bit with a novel idea, and publish your paper. Then
somebody else does that with your paper, and does the same.. at some
point, we will have a huge quantity of vibe coded code on github, and
two similar papers will have very different underlying implementations,
so hard to reason about and hard to change.
From a learning perspective, you try to understand the code, and it's
all spaghetti, and you loose more time understanding the code than it
would take to just reimplement it. You also learn a lot by not only
reading the paper but reading the authors code where most of the small
details reside.
And I'm not even talking about the reliability of the code, test to
know that it's the correct implementation. Authors try to make papers
as close as possible to the implementation but sometimes subtle steps
are removed, sometimes from inadvertance, sometimes because the number
of pages is lionmited.
A paper and an implementation are not one-to-one mappings
exe34 wrote 11 hours 54 min ago:
> you try to understand the code, and it's all spaghetti, and you
loose more time understanding the code than it would take to just
reimplement it.
I agree with you in general, but maybe the jump would be similar to
the one from hand-written punchcards/assembly to higher level
compilers. Very few people worry about the asm generated from GHC for
example. So maybe a lot of code would be like that. I also imagine at
some point a better intermediate language for LLMs to generate will
be discovered and suddenly that's how most programs will be written.
dadoomer wrote 4 hours 59 min ago:
> the jump would be similar to the one from hand-written
punchcards/assembly to higher level compilers
I wouldn't. Compilers are not stochastic text models and they can
be verified and reasoned about to a great extent.
somethingsome wrote 11 hours 16 min ago:
I would love that, I mostly work with ideas and the codes are
implementation details for me, so yes, in some way, having
automated code generation would allow me to be way more productive.
I'm not against it, I'm just scared about the efficiency of the
approach by an LLM (at the moment at least)
The example codes they give is 'implementing deep learning papers',
I find those papers the easiest to implement compared to some
obscure algorithm for example that can't rely on frameworks such as
pytorch and where speed is critical.
I can't find the essay, but I think it was wolfram that wrote that
we should let students use Mathematica and educate them in using it
from a young age, the rationale behind is: before you had to use
logarithmic tables, and it took much time during the education.
Then, with the event of the calculator, students could
instantaneously compute logarithms, so they could focus on more
advanced ideas that use them. With Mathematica they could
automatically execute matrix operations, so they would spend most
of the time thinking about matrix operations instead of just
learning how to manipulate a matrix by hand.
So with more powerful tools, you can expand the capabilities
faster.
But the main difference I see here, is that maths are precise and
well defined. Here you get a software which is a sample in the
space of possible softwares that solve the problem (if you are
lucky).
To get to the metaphorical point punchcards->GHC you need a LLM
tool that give always the same answer, and hopefully, the optimal
one, and with small changes in the paper, it moves the software in
the space of viable softwares only a bit. Maybe we will get there,
but this is not yet what this paper proposes
UltraSane wrote 10 hours 29 min ago:
It would be very interesting to teach kids math using Mathmatica
starting from kindergarten.
Narew wrote 12 hours 10 min ago:
Honestly, the code from my interns have greatly improve since they
use AI. And there is lots of really ugly and hard to read code from
papers. So I don't think it will be an obvious loss of readability to
have code completely generated by AI :)
somethingsome wrote 11 hours 27 min ago:
Very interesting, do you have a specific approach to educate them
in how to use LLMs? Or they do it free wheel? Do you give advices?
If so which kind?
I would love to have a structured approach to help students learn
to use better LLMs.
What I have observed (for uni students) is that they produce better
code overall, but have no idea how it works, and would not be able
to reproduce it without LLMs. (this is from first to last year)
Narew wrote 9 hours 51 min ago:
For the moment it's a bit free wheel. And I agree the code is
better but they could probably not reproduce it themself.
I honestly don't know how to "force" them to understand the code
the llm write if the code is clean. But this happen when the code
produce by llm is over complicated or bad and we catch that by
doing code review.
I have the impression it will create even more disparity between
student, students that just use llm code and the ones that try to
understand it.
tomrod wrote 19 hours 34 min ago:
> we will have a huge quantity of vibe coded code on github
That may actually be an improvement over much of the code that is
generated for papers.
sitkack wrote 22 hours 19 min ago:
I have had good results doing bidirectional programming in Tex <=>
Python.
somethingsome wrote 13 hours 15 min ago:
Can you give more details? I'm curious
sitkack wrote 4 hours 24 min ago:
Give it a try, have it teach you tex for the summation notation,
have it write the code, modify the code, have it translate back to
tex. Repeat.
You can do a quick test to see which models have been trained on
tex.
Keep a tex visualizer handy.
bjourne wrote 22 hours 32 min ago:
It relies on OpenAI's o3-mini model which (I think) you have to pay
for.
colkassad wrote 1 day ago:
It would be neat to run their pdf through their implementation[1] and
compare results.
URI [1]: https://github.com/going-doer/Paper2Code
polygot wrote 13 hours 25 min ago:
I decided to do that, and made Paper2Code2Code:
URI [1]: https://github.com/alexyorke/Paper2Code2Code/tree/main
IshKebab wrote 8 hours 51 min ago:
And... does it work?
protolyticmind wrote 15 hours 34 min ago:
I thought it would be humorously ironic :D
endofreach wrote 22 hours 30 min ago:
Damn, i was hoping the link was your result of that. Please do that.
I can't start another project currently. But i'd love the short
result as an anecdote. But if you don't do it, i might have to.
Please let me know. Great idea, really.
omneity wrote 22 hours 27 min ago:
If I was the paper author I would have done it and include the
results as an appendix or a repo.
JackYoustra wrote 22 hours 19 min ago:
haha would that itself be a product of the paper then?
omneity wrote 22 hours 18 min ago:
Maybe by doing it enough times o3-mini will end up
reimplementing itself?
endofreach wrote 21 hours 50 min ago:
Imagine, this was actually the consequenceâ against all
odds. The true power of AI is about to be discovered...
through this silly experiment... and you are the one... all
you gotta doâ is do it. And imagine you don't do it,
because you think it can't lead to such a serious result...
and if we miss this great leap forward... go, throw your life
away and do this. now. the universe is waiting on you, my
friend.
barotalomey wrote 6 hours 49 min ago:
That's how nothing ever works outside fiction.
DIR <- back to front page