{"id":168450,"date":"2021-06-09T15:15:30","date_gmt":"2021-06-09T10:15:30","guid":{"rendered":"https:\/\/venturebeat.com\/?p=2694904"},"modified":"2021-06-09T15:15:30","modified_gmt":"2021-06-09T10:15:30","slug":"facebook-proposes-nethack-as-a-grand-challenge-in-ai-research","status":"publish","type":"post","link":"https:\/\/www.technologyforyou.org\/facebook-proposes-nethack-as-a-grand-challenge-in-ai-research\/","title":{"rendered":"Facebook proposes NetHack as a grand challenge in AI research"},"content":{"rendered":"
\n

Elevate your enterprise data technology and strategy at Transform 2021<\/a><\/em>. <\/p>\n

\n<\/div>\n

Facebook today proposed NetHack as a grand challenge for AI research, for which the company is launching a competition at the NeurIPS 2021 AI conference in Sydney, Australia. It\u2019s Facebook\u2019s assertion that NetHack, an \u201980s video game with simple visuals that\u2019s considered among the hardest in the world, can enable data scientists to benchmark state-of-the-art AI methods in a complex environment without the need to run experiments on a powerful computer.<\/p>\n

Games have served as AI benchmarks for AI for decades, but things really kicked into gear in 2013 \u2014 the year Google\u2019s DeepMind demonstrated a system that could play Pong, Breakout, Space Invaders, Seaquest, Beamrider, Enduro, and Q*bert at superhuman levels. The advancements aren\u2019t merely improving game design, according to experts like DeepMind cofounder Demis Hassabis. Rather, they\u2019re informing the development of systems that might one day diagnose illnesses, predict complicated protein structures<\/a>, and segment CT scans<\/a>.<\/p>\n

In particular, reinforcement learning<\/a> \u2014 a type of AI that can learn strategies to orchestrate large systems like manufacturing plants, traffic control systems, financial portfolios, and robots \u2014 is transitioning from research labs to highly impactful, real-world applications. For example, self-driving car companies like Wayve<\/a> and Waymo<\/a> are using reinforcement learning to develop the control systems for their cars. And via Microsoft\u2019s Bonsai<\/a>, Siemens is employing reinforcement learning to calibrate its CNC machines.<\/p>\n

\u201cRecent advances in reinforcement learning have been fueled by simulation environments such as games like StarCraft II, Dota 2, or Minecraft. However, this progress came at substantial computational costs, often requiring running thousands of GPUs in parallel for a single experiment, while also falling short of leading to \u2026 methods that can be transferred to more real-world problems outside of these games,\u201d Facebook AI researchers Edward Grefenstette, Tim Rockt\u00e4schel, and Eric Hambro wrote in a blog post. \u201cWe need environments that are complex, highlighting shortcomings of RL, while also allowing extremely fast simulation at low computation costs.\u201d<\/p>\n

NetHack<\/h2>\n

Facebook\u2019s proposal follows the release of the company\u2019s NetHack Learning Environment (NHLE)<\/a>, a research tool based on the original NetHack. (The NetHack Challenge is in turn based on the NHLE.) NetHack, which was first released in 1987, tasks players with descending more than 50 dungeon levels to retrieve a magical amulet, during which they must use wands, weapons, armors, potions, spellbooks, and other items and fight monsters. Levels in NetHack are procedurally generated and every game is different, which the Facebook researchers note tests the generalization limits of leading AI.<\/p>\n

\u201cWinning a game of NetHack requires long term planning in an incredibly unforgiving environment. Once a player\u2019s character dies \u2026 the game starts from scratch in an entirely new dungeon,\u201d Grefenstette, Rockt\u00e4schel, and Hambro continued. \u201cSuccessfully completing the game as an expert player takes on average 25 to 50 times more steps than an average StarCraft II game, and players\u2019 interactions with objects and the environment are extremely complex, so success often hinges on calling upon imagination to solve problems in creative or surprising ways as well as consulting external knowledge sources [such as] the official NetHack Guidebook<\/a>, the NetHack Wiki<\/a>, and online videos and forum discussions].\u201d<\/p>\n

\"Facebook<\/p>\n

Partial observation makes exploration in NetHack essential, and procedural generation and \u201cpermadeath\u201d make the cost of failure significant. And AI can\u2019t reset or interfere with the environment, making the methods that underpin systems like DeepMind\u2019s AlphaZero<\/a> for StarCraft II or Uber\u2019s Go-Explore<\/a> for Montezuma\u2019s Revenge impossible.<\/p>\n

\u201c[The challenges in NetHack] range from randomized mazes to more structured challenges, like large rooms full of monsters and traps, towns and forts, and hazards such as kraken-infested waters,\u201d Grefenstette, Rockt\u00e4schel, and Hambro said. \u201cNew ways of dealing with the ever changing observations in a stochastic and rich game world calls for the development of techniques that have a better chance of scaling to real-world settings with high degrees of variability.\u201d<\/p>\n

Lightweight<\/h2>\n

NetHack has another advantage in its lightweight architecture. A turn-based, ASCII-art world and a game engine written primarily in C captures its complexity. NetHack forgoes all but the simplest physics while rendering symbols instead of pixels, importantly, allowing AI to learn quickly without wasting computational resources on simulating dynamics or rendering observations.<\/p>\n

Indeed, training sophisticated machine learning models in the cloud remains prohibitively expensive. According to a recent Synced report<\/a>, the University of Washington\u2019s Grover, which is tailored for both the generation and detection of fake news, cost $25,000 to train over the course of two weeks. OpenAI racked up $256 per hour to train its GPT-2<\/a> language model, and Google spent an estimated $6,912 training BERT<\/a>, a bidirectional transformer model that redefined the state of the art for 11 natural language processing tasks.<\/p>\n

By contrast, a single high-end graphics card is sufficient to train AI-driven NetHack agents hundreds of millions of steps a day using the TorchBeast framework, which supports further scaling by adding more graphics cards or machines. Agents can experience billions of steps in the environment in a reasonable time frame while still challenging the limits of what current techniques can achieve.<\/p>\n

\"Facebook<\/p>\n

\u201c[The NHLE] can train reinforcement learning agents \u202615 times faster than even decade-old Atari benchmark[s]. Furthermore, NetHack can be used to test the limits of even more recent state-of-the-art deep reinforcement learning methods while running 50 to 100 times faster than challenges of comparable difficulty while providing a higher degree of complexity.\u201d<\/p>\n

Challenge<\/h2>\n

The NHLE consists of three components: a Python interface to NetHack using the popular OpenAI Gym API, a suite of benchmark tasks, and a baseline machine learning agent. To beat the NetHack Challenge, entrants must develop AI that can reliably either win at NetHack or achieve as high a score as possible. In doing so, the competition aims to yield a head-to-head comparison of different methods and new benchmarks for future research, while at the same time showcasing the suitability of the NHLE as a setting for research.<\/p>\n

There won\u2019t be restrictions on how the systems can be trained for the NetHack Challenge, Facebook says \u2014 participants are welcome to use techniques besides machine learning if they choose. Awards will be given for (1) the best overall AI system, (2) the best AI system not using a neural network<\/a>, and (3) the best AI system from an academic or independent team.<\/p>\n

Grefenstette, Rockt\u00e4schel, and Hambro say that achieving these objectives will lay the groundwork for follow-up competitions focused on specific aspects of AI. Moreover, the NetHack Challenge might help bring light to classes of training methods and modeling approaches capable of dealing with highly varied environments and a high cost of errors, like having to restart from scratch if a character is killed by a creature.<\/p>\n

\u201cMany real-world and industrial problems \u2014 navigation, for example \u2014 share these characteristics. Consequently, making progress in NetHack is making progress toward reinforcement learning in a wider range of applications,\u201d Grefenstette, Rockt\u00e4schel, and Hambro said.<\/p>\n

Facebook\u2019s NeurIPS 2021 NetHack Challenge will be conducted in partnership with co-organizer AIcrowd<\/a>, and it\u2019ll run from early June through October. The winners will be announced at NeurIPS in December.<\/p>\n

\n

GamesBeat<\/h3>\n

GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.<\/span> How will you do that? Membership includes access to:<\/span><\/p>\n