{"id":168626,"date":"2021-06-09T22:01:04","date_gmt":"2021-06-09T17:01:04","guid":{"rendered":"https:\/\/venturebeat.com\/?p=2695704"},"modified":"2021-06-09T22:01:04","modified_gmt":"2021-06-09T17:01:04","slug":"eleutherai-claims-new-nlp-model-approaches-gpt-3-level-performance","status":"publish","type":"post","link":"https:\/\/www.technologyforyou.org\/eleutherai-claims-new-nlp-model-approaches-gpt-3-level-performance\/","title":{"rendered":"EleutherAI claims new NLP model approaches GPT-3-level performance"},"content":{"rendered":"
\n

Elevate your enterprise data technology and strategy at Transform 2021<\/a><\/em>. <\/p>\n

\n<\/div>\n

AI-powered language systems have transformative potential, particularly in the enterprise. They\u2019re already being used to drive chatbots, translate natural language into structured query language, create application layouts and spreadsheets, and improve the accuracy of web search products. Perhaps the best-known AI text-generator, OpenAI\u2019s GPT-3<\/a>, is being used in more than 300 different apps by tens of thousands of developers and producing 4.5 billion words per day.<\/p>\n

As interest in AI rises in business, advisory firm Mordor Intelligence forecasts that the natural language processing (NLP) market will more than triple<\/a> its revenue by 2025. But noncommercial, open source efforts are concurrently gaining steam, as evidenced by the progress made by EleutherAI<\/a>. A grassroots collection of AI researchers, EleutherAI this week released GPT-J-6B (GPT-J), a model the group claims performs nearly on par with an equivalent-sized GPT-3 model on various tasks.<\/p>\n

\u201cWe think it\u2019s probably fair to say this is currently the best open source autoregressive language model you can get by a pretty wide margin,\u201d Connor Leahy, one of the founding members of EleutherAI, told VentureBeat.<\/p>\n

GPT-J is what\u2019s known as a Transformer<\/a> model, which means it weighs the influence of different parts of input data rather than treating all the input data the same. Transformers don\u2019t need to process the beginning of a sentence before the end. Instead, they identify the context that confers meaning to a word in the sentence, enabling them to process input data in parallel.<\/p>\n

The Transformer architecture forms the backbone of language models including GPT-3 and Google\u2019s BERT<\/a>, but EleutherAI claims that GPT-J took less time to train compared with other large-scale model developments. They researchers attribute this to the use of Jax, DeepMind\u2019s Python library designed for machine learning research, as well as training on Google\u2019s tensor processing units (TPU)<\/a>, application-specific integrated circuits (ASICs) developed specifically to accelerate AI.<\/p>\n

Training GPT-J<\/h2>\n

EleutherAI says that GPT-J contains roughly 6 billion parameters, the parts of the machine learning model learned from historical training data. It was trained over the course of five weeks on 400 billion tokens from a dataset created by EleutherAI called The Pile, a 835GB collection of 22 smaller datasets including academic sources (e.g., Arxiv, PubMed), communities (StackExchange, Wikipedia), code repositories (Github), and more. Tokens are a way of separating pieces of text into smaller units in natural language, and they can be words, characters, or parts of words.<\/p>\n

\n
\"EleutherAI\"<\/p>\n

Above: GPT-J can solve basic math problems.<\/p>\n

Image Credit: EleutherAI<\/em><\/p>\n<\/div>\n<\/div>\n

For compute, EleutherAI was able to leverage the TPU Research Cloud, a Google Cloud initiative that supports projects with the expectation that the results of the research will be shared via code and models. GPT-J\u2019s code and the trained model are open sourced under the MIT license and can be used for free using HuggingFace\u2019s Transformers platform<\/a> or EleutherAI\u2019s website.<\/p>\n

GPT-J is more capable than the two models EleutherAI previously released, GPT-Neo 1.3B and GPT-Neo 2.7B<\/a>. For example, it can perform addition and subtraction and prove simple mathematical theorems, like \u201cAny cyclic group is abelian.\u201d It can also answer quantitative reasoning questions from a popular test dataset (BoolQ) and generate pseudocode.<\/p>\n

\n
\"EleutherAI\"<\/p>\n

Above: GPT-J proving a theorem.<\/p>\n

Image Credit: EleutherAI<\/em><\/p>\n<\/div>\n<\/div>\n

\u201c[OpenAI\u2019s] GPT-2 was about 1.5 billion parameters and doesn\u2019t have the best performance since it\u2019s a bit old, GPT-Neo was about 2.7 billion parameters but somewhat underperforms equal-sized GPT-3 models, GPT-J, the new one, is now 6B \u2014 sized similar to the Curie model of OpenAI, we believe,\u201d Leahy said.<\/p>\n

Looking ahead<\/h2>\n

EleutherAI plans to eventually deliver the code and weights needed to run a model similar, though not identical, to the full \u201cDaVinci\u201d GPT-3. (Weights are parameters within a neural network that transform input data.) Compared with GPT-J, the full GPT-3 contains 175 billion parameters and was trained on 499 billion tokens from a 45TB dataset.<\/p>\n

Language models like GPT-3 often amplify biases encoded in data. A portion of the training data is not uncommonly sourced from communities with pervasive<\/a> gender, race, and religious prejudices. OpenAI notes that this can lead to placing words like \u201cnaughty\u201d or \u201csucked\u201d near female pronouns and \u201cIslam\u201d near words like \u201cterrorism.\u201d Other studies, like one published in April by Intel, MIT, and the Canadian Institute for Advanced Research (CIFAR) researchers, have found high levels of stereotypical bias in some of the most popular models.<\/p>\n

\n
\"EleutherAI\"<\/p>\n

Above: GPT-J answering a word problem.<\/p>\n

Image Credit: EleutherAI<\/em><\/p>\n<\/div>\n<\/div>\n

But EleutherAI claims to have performed \u201cextensive bias analysis\u201d on The Pile and made \u201ctough editorial decisions\u201d to exclude datasets they felt were \u201cunacceptably negatively biased\u201d toward certain groups or views.<\/p>\n

While EleutherAI\u2019s model might not be cutting edge in terms of its capabilities, it could go a long way toward solving a common problem in tech: the disconnect between research and engineering teams. As Hugging Face CEO Cl\u00e9ment Delangue told VentureBeat in a recent interview, on the one hand, tech giants provide black-box NLP APIs while also releasing open source repositories that can be hard to use or aren\u2019t well-maintained. EleutherAI\u2019s efforts could help enterprises to realize the business value of NLP without having to do much of the legwork themselves.<\/p>\n

\n

VentureBeat<\/h3>\n

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact. Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:<\/p>\n