This seems very useful. Why not make it paid ? Do you think your customers won’t buy ? Have you tried ?

What would your customers need to make them want to pay for it ?

I think most of the people who sign up for email alerts would never pay. Lots of them are indie hackers or folks with a side project - I've been there, and know how price sensitive those communities are. I'd rather they use the service for free than not at all - I get valuable feedback from that, a marketing boost if they tell others about it, and the validation of having built something other people use.

I do have a paid plan for people who want Slack notifications, and I think those folks ought to be happy to pay. My hope is that I'll eventually get a few paid signups and that those will cover the costs of the service (which are minimal).

I know I lose a bit of revenue with the above approach, but it's a tradeoff I'm happy to make.

Do you plan to write the book ? I’d like to read it

Yes, please write it. I will buy and read it.

I'll buy it, read it and take notes.

IMG_0163 - Gold

nopes.. beyond starter projects, raw usage of chatgpt hardly leads to any gains in productivity in the enterprise setting .

So far no, but in the future with more specific and enterprise suitable tooling - likely

Only murmurs for now, admittedly, but I heard a reason IBM is training models (e.g. https://research.ibm.com/blog/granite-code-models-open-sourc...) is because they provide LLM-based systems for enterprise customers to work on ancient, legacy codebases in languages like COBOL more easily. If true, I could definitely see how that might boost productivity as fewer people remain fully trained in the details of such old systems and languages.

That sounds a lot like their old Watson claims.

unless IBM beats GH copilot enterprise that's by itself not that great, hard doubt it'll make a big impact

I don't have any insider knowledge, but maybe IBM could get its hands on a lot more legacy code for weird arcane systems than GitHub could? It would make their models more specific than those trained on 50% Python at least.

That would be Github that's owned by Microsoft who also make Windows? They can get weird arcane code.

maybe, I've used copilot enterprise in our company for a big C++ project and to say I was unimpressed is an understatement

Lots of old code not on GH that they could combine with what is on GH…

Yes, but then a PhD is a chicken and egg problem. You can't get into the top PhD programs under a respectable professor - unless you have some publications under your name.

So how does one get published, enough to get into a good PhD program?

No, I meant to ask HN, but didn’t realise that if I put a URL it doesn’t start a discussion.. and now there is no delete button

> Other tasks have an element of looping which is really fundamental in computer science

Have you tried any of these agent based workflows that claim to be able to loop with some amount of reasoning ?

No, but I have read many papers where people have some success but don't really get it to work consistently.

From Godel, Escher, Bach and my own experience I'm afraid that's a way to spend the next 1000 years pushing a bubble around under a rug.

If you had something that fully automated the cycle of running the compiler and running tests at least I wouldn't have the tedium of doing all that, particularly the bit about having to reprompt with the last compiler error.

Right. I think the ability to run tests/compilation as a feedback mechanism is definitely interesting .

I wonder whether the problem is that its landed on an incorrect output sample (i.e re-sampling another code output would fix it ) .. vs … a feedback loop on the compiler output is what it really needs ..

100% I’d rather have specialised LLMs that can solve some hairy problems like CSS , than what we have now

I feel like CSS is more of a problem for a SAT solver than for an LLM. But really it is the place where I feel need the most help, particularly when debugging or modifying CSS written by people who didn't really understand CSS. (With a total of 40+ standards documents defining it, who really does?)

Precisely. I think it helps with smaller/mundane tasks (that it has seen in its training), but the tasks that actually require a higher level reasoning and understanding of the bigger picture - are not something we can expect the current LLM's to do.

However as I was researching, there are a few interesting ideas in this space that might help these LLM's solve more complex problems in the future. Post here if interested: https://kshitij-banerjee.github.io/2024/04/30/can-llms-produ...

After using code LLMs' for a bit, I wonder whether there is significant productivity gain in using them. I found that its limited for me.

Main reason: a) to find the right code, I have to keep sampling it, and b) it doesn't seem to be able to solve larger / more complex problems that I actually find more need for.

I found some interesting research on combining planning-algorithms for complex problems, and some ideas on guiding the LLM's decoding process towards correctness by optimizing it via reward functions and reducing the search space. I've detailed and summarised the main points in the post above.

Questions:

1) Do you find code LLM's really useful? Please share some stories / examples where they help vs they didn't. I'm trying to form a better understanding they are just fancy, vs actually productive and useful to most

2) Any other research ideas being pursued in this field ? / what are you trying ?