JungleGym AI | A playground for testing and developing autonomous agents with AI models (2023)
JungleGym is an open source playground for testing and developing autonomous agents with different agent datasets and benchmarks, like WebArena, Mind2Web and AgentInstruct. The playground has realistic, fully functional, sandboxed web sites, benchmarks and an API, all in a single environment.
On October 2024, mostly as a fun side project, a friend (Andres Campero) and I worked for ~2.5 weeks on making a submission with a custom set of trained models to the ARC challenge competition on Kaggle to see how far we could push AI models on this benchmark.
Currently, we are in position 14 out of 1427 teams. I plan to keep working on this (still mostly for fun), but training these models can get a bit pricey.
The prized competition has some requirements: the model(s) need to fit in a single P100 GPU (16Gb) or in two T4s (16Gb each), and a maximum running time of 12 hours for all the tasks and the solution needs to be completely offline, that means no GPT4, Claude, API calls, etc. So it only leaves you with a few options of pre-trained models you could use.
Some approaches:
- Inference-time search
- Data augmentation and synthetic data of known puzzles (transduction)
- CoT traces on multiple solutions (positive and negative) with grids (transduction)
- CoT traces on multiple solutions (positive and negative) with code synthesis (induction)
- Long training runs (many epochs) over synthetic data
- Multiple models for overall solution (induction and transduction)
- Active inference (train model on the fly)
- Used Llama and Qwen models
Llama2.ai | The first llama2 chatbot interface (2023)
Built and deployed Llama2.ai, the first AI chatbot interface for the Llama2 models, which became the most popular interface for the Llama2 models when Mark Zuckerberg announced them. Reached over 3000+ concurrent requests. The system was built with on 100 load-balanced servers and hundreds of GPUs (H100s). (Llama2.ai is now run by Replicate)
AnyPod.ai | Semantic Search Engine for Youtube/Podcasts (2022)
A weekend project to search for phrases, ideas or semantic questions on YouTube/Podcasts. Used Whisper, an embedding model (all-mpnet-base-v2) with Sentence Transformers and FAISS to return top k 20 results from multiple simultaneous YouTube channels.
arXivGPT | Twitter bot that posts the most relevant/trending AI papers on arXiv.org (2023)
Built arXivGPT originally for myself while trying to filter through and read the most relevant AI papers on arXiv.org, as there was a lot of noise. ArXivGPT creates summaries of trending AI papers on arXiv (using a set of heuristics), listing the most important points and authors.
YouTranscription | Automated Whisper transcripts from YouTube channels (2022)
A weekend project to Automate transcriptions of YouTube videos or channels into searchable transcripts.
LoweBot | An autonomous customer service and inventory management Robot (2016)
LoweBot is a robot that was deployed across multiple retailers, including Lowe's Home Improvement stores. It's an autonomous customer service robot that helps customers find items in stores, understanding multiple languages, and it scans inventory using computer vision overnight.
SIMPL (2016)
A custom trained AI model to allow an easy way to program industrial robot arms with no coding required, only with drawing bounding boxes over objects in a easy to use interface. It uses a custom trained single-shot CNN trained on thousands of objects running locally.
AI model for weapon detection (2022)
A trained AI model to detect weapons from low resolution security cameras running in real-time (~25fps) in constrained compute (optimized model for local hardware) and send SMS alerts.
Covid-19 Ventilator (2020)
Built an open-source Covid ventilator due to the shortage of ventilators in 2020, which helped over 200 teams around the world building and deploying ventilators. The first version of this was running on a Raspberry Pi. Later versions used a custom PCB with an ARM chip.
Contact Tracing by precise proximity with Ultra-wideband (2020)
Created a contact traicing device using precise distance measurement using Ultra-wideband (UWB). The contact tracer allows people to socially distance (by an haptic feedback) and keep track of every contact within 10 feet precisely.
Vision-based, autonomous wheelchair robot (2011)
Developed an autonomous wheelchair robot (KuruRobo) using VSLAM for disabled people while I was doing Computer Vision research at the Kanazawa Institute of Japan.
Other random ones in Japan
