Codalab faced a significant limitation in its support for programming languages, primarily accommodating only Python submissions, which restricted its appeal to a broader AI and machine learning audience. This limitation, coupled with the need for enhanced functionality for handling machine learning competitions, prompted CKC's involvement. CKC addressed these challenges by integrating Docker container technology, which enabled the platform to support a variety of programming languages like Julia, R, and .NET. This integration not only expanded Codalab's usability for a wider range of users but also enhanced its system stability, security, and capability to cater to both educational and research-oriented applications, thereby elevating its overall functionality and utility in the AI community.
We architected a solution with Docker containers on Codalab workers using Signed URLs for saving solutions and grabbing submission metdata. Docker Volumes would contain relevant datasets for the appropriate phase of a competition, and save the results to be compared to other submissions. We even made various plugins for allowing Artificial Intelligence professors to compare submissions privately and programmatically. We used Docker-in-Docker for the workers so a privileged docker container could spawn children and destroy them safely after completion or error. Workers could run for months without problems!