All of Bruce G's Comments + Replies

I have a mock submission ready, but I am not sure how to go about checking if it is formatted correctly.

Regarding coding experience, I know python, but I do not have experience working with typescript or Docker, so I am not clear on what I am supposed to do with those parts of the instructions.

If possible, It would be helpful to be able to go through it on a zoom meeting so I could do a screen-share.

1Beth Barnes
Hey! It sounds like you're pretty confused about how to follow the instructions for getting the VM set up and testing your task code. We probably don't have time to walk you through the Docker setup etc - sorry. But maybe you can find someone else who's able to help you with that? 

Thanks for your reply. I found the agent folder you are referring to with 'main.ts', 'package.json', and  'tsconfig.json', but I am not clear on how I am supposed to use it. I just get an error message when I open the 'main.ts' file:

Regarding the task.py file, would it be better to have the instructions for the task in comments in the python file, or in a separate text file, or both? Will the LLM have the ability to run code in the python file, read the output of the code it runs, and create new cells to run further blocks of code?

And if an automated ... (read more)

1Beth Barnes
Did you try following the instructions in the README.md in the main folder for setting up the docker container and running an agent on the example task? I think your computer is reading the .ts extension and thinking it's a translation file: https://doc.qt.io/qt-6/linguist-translating-strings.html But it's actually a typescript file.  You'll need to open it with a text editor instead. Yeah, doing a walkthrough of a task submission could be great. I think it's useful if you have a decent amount of coding experience though - if you happen to be a non-coder there might be quite a lot of explaining required.  

If anyone is planning to send in a task and needs someone for the human-comparison QA part, I would be open to considering it in exchange for splitting the bounty.

I would also consider sending in some tasks/ideas, but I have questions about the implementation part.

From the README document included in the zip file:

## Infra Overview

In this setup, tasks are defined in Python and agents are defined in Typescript. The task format supports having multiple variants of a particular task, but you can ignore variants if you like (and just use single variant named fo

... (read more)
1Beth Barnes
Thanks a bunch for the detailed questions, helpful! 1. Re agents - sorry for the confusing phrasing.  A simple agent is included in the starter pack, in case this is helpful as you develop + test your task. 2. Submission:  You need to submit the task folder containing any necessary resources, the yourtask.py file which does any necessary setup for the task and implements automatic scoring if applicable, and the filled out README. The README has sections which will need you to attach some examples of walkthroughs / completing the task. Any suggestions for making this clearer? The existing text is: