DML was established in 1987 to manufacture plastic Model Kits according to its own designs, and then export them to all parts of the globe via agents in different countries. The high quality products that DML have produced have won numerous awards from magazines and modeling societies throughout the world. The products are marketed under the label "Dragon". Since its establishment, DML has constantly been investing capital in new equipment and tools. Since its founding it has now released more than 1,000 items. Recent years have seen DML's products marketed under the label "Revell/Monogram" in the USA, "Revell" and "Italeri" in Europe, and "Hasegawa" and "GSI" in Japan, all ODM products.
In 1997, DML diversified into the airliner model field, manufacturing airliner models for airline promotional use and for collectors. The resulting Dragon Wings line of 1/400 scale aircraft has since grown to become one of DML's most successful and diverse series, representing the unique livery of over 100 commercial and national airlines ... a total that is still climbing. DML maintains strict standards of quality which reflect the original specifications of the aircraft, feature working parts, accurate commercial markings and scale. Dragon Wings continues to hold a unique and respected position in the die-cast model aircraft industry. DML is the officially appointed supplier of aircraft models for Airbus and Boeing for their promotional and souvenir purposes.
In 1999, DML brought its modeling background and expertise to 1/6 scale with the introduction of its New Generation Life Action Figure series. By January of the following year, this series - which features fully posable military and licensed 12-inch figures with meticulously researched cloth uniforms, detailed weapons and equipment -- was universally recognized as setting the new benchmark in high-quality collectable action figures. This range has won many awards from industry magazines and has attracted a strong and passionate base of Dragon action figure collectors worldwide. The series currently stands at some 400-plus individual releases, with subjects varying from World War II, Modern Special Operations and Law Enforcement, to licensed character figures from movies, popular music, sports, electronic games and comics. This line of 12-inch action figures is arguably of the best quality in the world.
DML continued to expand its catalog of collector-oriented figures in the new millennium with its "Dragon Minis" series -- smaller and more stylized figures that are geared towards a younger audience with high emphasis on both collectability and a "fun factor". Another new range was the "Action18" series of ultra-realistic 1/18 scale fully posable action figures - a unique composite of modeling and action figure philosophies, but at a scale more appropriate for modelers and diorama builders.
The continued success of Dragon Wings led to an entirely new series of die-cast aircraft models for DML in 2002 - the Warbirds - an ever-expanding line of 1/72 scale fighter aircraft ranging from WWII classics to the cutting-edge planes of the modern day. With an emphasis on accuracy and peerless quality, this series has become a market leader in the world of aircraft collectables.
For 2003, DML ventured into yet more uncharted ground with "Micro X Tech" and the CAN.DO series of collectable trading miniatures. "Micro X Tech" is a brand-new Palm Top R/C Armor series in multiple scales, combining the real function of R/C with the versatility of modeling. The CAN.DO line of trading miniatures became a hit with Japan teenagers when it debuted early in 2003, and they sold like hot cakes in Japanese convenience and chain stores. This series brings back the old-style fun of collecting by reintroducing an element of surprise! CAN.DO collectors can choose their subjects from amongst several highly-detailed miniature series like World Soccer, Pocket Armor, Moomin, and more.
2003 also saw DML introducing yet another line of die-cast models - Dragon Armor. Compatible in scale with DML's Warbirds series of fighter aircraft, Dragon Armor adds just the right amount of action to 1/72 scale battlefields! Pre-assembled, pre-painted, and epitomizing DML's tradition of fine detail, superb quality, and accuracy, Dragon Armor immediately became a big hit with modelers and collectors alike, and has set an example for others to follow.
2005 has heralded "the dawn of the golden age of modeling", to quote modelers themselves. During this year DML has inaugurated a quantum leap forward by issuing kits that utilize revolutionary technology and that are engineered down to the smallest detail. In doing so DML has set a new standard in the model kit industry. It does this by offering well researched, multi-material, innovatively designed, value-for-money model kits that leave modelers with a jaw-dropping sense of awe! DML continues to strive to fulfill its motto "Steps Ahead...Always!" and things will not end there. DML will continue to develop new technology, coupling it with innovative design to maintain their market leadership.
Currently, DML's business scope includes plastic models, action figures, die-cast collectables, resin, PVC and vinyl, and R/C products. Dragon sets the industry standard in many areas, and is a market leader in many of its lines. Its fine products are much sought after by collectors on ebay and suchlike, and truly, DML has set an example for other manufacturers to emulate.
Foundation models are models that are pretrained from raw data at scale and can be adapted to various downstream applications. For example, language models, like BERT and GPT family, are pretrained from a large amount of raw text such as Wikipedia and Books, and the resulting models can be adapted (e.g. via finetuning, prompting) to help an extremely wide range of applications, including question answering, text classification, etc. These language models achieve remarkable performance on many natural language processing (NLP) tasks, becoming the foundation of today’s NLP systems. They are serving important roles in various products and tools that we use every day, such as search engines like Google and personal assistants like Alexa.
What can foundation models learn from?
While text is commonly used to train language models, knowledge graphs (KG) provide complementary information to text. KGs offer structured background knowledge by representing entities as nodes and relations between them as edges, e.g. <Leonardo da Vinci — born in — Italy>. Examples of knowledge graphs include Freebase, Wikidata (general-purpose facts), ConceptNet (commonsense), and UMLS (biomedical facts).
Text and KGs have complementary strengths. Text has a broad coverage of knowledge and captures rich context. Meanwhile, KGs are structured and offer scaffolds for logical or multi-step reasoning by providing paths between entities. Some KGs also include knowledge that may not be commonly stated in text; for instance, people do not often state obvious facts like “people breathe” or compositional sentences like “The birthplace of the painter of the Mona Lisa is Italy”. Hence, text and KG have the potential to mutually inform each other, i.e. text can contextualize the KG with its rich prose, and KG can ground the text with its structure for reasoning.
Training a foundation model from text and KG
The above observation motivates research in fusing the strengths of the two modalities, text and KG. In our recent work published at NeurIPS 2022, we develop DRAGON, a new method to train a foundation model jointly from text and KG.
Challenges
To pretrain a powerful foundation model, we need both (i) an expressive model that allows the two modalities to interact in a deep, bidirectional manner; and (ii) a self-supervised training objective that allows the model to learn joint reasoning over text and KG at scale without manual labels. Existing models that combine text and KGs tend to perform indirect/uni-directional fusion of the two modalities, or are supervised by small labeled datasets rather than being self-supervised at scale. We will address these challenges below.
We introduce DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised method to pretrain a deeply fused foundation model from text and KG. As an overview, DRAGON consists of three steps. We first sample pairs of text segments and relevant KG subgraphs to create inputs for the model (Step 1). We then use a deep bidirectional model to fuse the input text and KG (Step 2), and finally pretrain the model using a joint self-supervised task over the two modalities (Step 3).
Step 1: Text-KG Input Sampling
Given a text corpus (e.g. Books) and a large KG (e.g. ConceptNet) as raw data, we want to sample informative pairs of (text segment, KG subgraph) as inputs for the model so that the text and KG are semantically related and can inform each other. To achieve this, for each text segment sampled from the text corpus, we retrieve a relevant subgraph from the KG by doing simple entity linking, i.e. string-match entities mentioned in the text segment to the KG, and extract these entities nodes as well as their neighbor nodes from the KG. Consequently, we obtain a pair of (text segment, local KG). Henceforth, we use “KG” to refer to this local KG for convenience.
Step 2: Deep Bidirectional Cross-Modal Model
Given the input text and KG, we want to design a model that can capture rich interactions between them. Some inspirations we can draw are deep bidirectional contextualization of inputs, which have made BERT very successful, and graph neural networks (GNNs), which are shown to be effective for modeling graph algorithms including knowledge graph reasoning. With these motivations, we designed a model architecture called GreaseLM that combines Transformer and GNN to fuse text and KG bidirectionally for multiple layers. Specifically, each layer of this model has a Transformer that encodes the input text and a GNN that encodes the input KG, which are then fused by a bidirectional modality interaction module.
Step 3: Bidirectional Self-supervision
The final step is to pretrain the model using the inputs we prepared. We train the model by unifying two self-supervised reasoning tasks over text and KG. The first task is language modeling, which predicts masked words or next words from the input text. The other task is link prediction, which predicts edges that were held out from the input KG. The intuition is that by combining the two tasks, the model is encouraged to use both the text and the KG to reason about the missing words in text and missing links in the KG. This joint training facilitates the model to propagate information bidirectionally between the two modalities.
We pretrain DRAGON in two domains
DRAGON improves over vanilla language models (LMs) and previous LM+KG models
We finetune and evaluate the pretrained DRAGON on diverse downstream tasks in each domain:
We compare DRAGON with two types of baselines. The first is a vanilla language model (LM), i.e. RoBERTa or BioLinkBERT. The second is GreaseLM, which takes a vanilla language model and *finetunes* it using KG, but it does not pretrain using KG. So the key difference is that DRAGON pretrains a language model using KG as well as finetuning.
The figure below shows the evaluation results. DRAGON outperforms the baseline language models and GreaseLM across the commonsense and biomedical question answering tasks. In particular, we can decompose and see the effect of using KG in pretraining with respect to the vanilla language model (purple arrow), and the effect of self-supervised pretraining with respect to GreaseLM (blue arrow). We can see that both components of DRAGON contribute to significant performance improvements.
Effective for complex reasoning
We also find that DRAGON exhibits several interesting strengths. The first strength is improved performance for complex reasoning tasks. We identified several types of complex reasoning such as questions that contain negation terms (like “no”, “never”) or conjunction terms (like “and”, “but”)—these indicate logical reasoning—and also questions that contain many prepositional phrases or many entity mentions—these indicate involving more reasoning constraints or steps. We find that DRAGON attains large performance gains on these complex questions compared to the baseline models.
The intuition is that because DRAGON is pretrained using KG, it learns to use KG as a scaffold for performing structured reasoning about entities. For instance, when the question contains a conjunction (example on the left of the figure), the model exhibits stronger attention weights over the entities related to the conjunction after several layers of the GNN over the KG, which lead to the correct answer. Similarly, when the question further contains a negation (example on the right of the figure), the model exhibits stronger attention weights over the entities that are not negated. One interpretation of these findings is that DRAGON uses the structure of KG and GNN as a scaffold for performing complex reasoning—this insight is related to recent works that provide language models with scratch space for doing intermediate reasoning. Another interpretation is that the GNN component of DRAGON learns to perform soft execution of natural language inputs (questions) on the KG—this insight is related to recent works showing that GNNs can learn to execute graph algorithms, including execution of complex logical queries on KG.
DRAGON also exhibits an ability to extrapolate to more complex questions. For instance, it adjusts the entity attention weights and final predictions accordingly when an extra context (i.e. extra reasoning step) is added to the original question, as in the figure below (left → right). Meanwhile, vanilla language models (RoBERTa) and KG-augmented finetuning (GreaseLM) struggle on these QA examples. This may suggest that KG-augmented pretraining (DRAGON) is important for acquiring broader reasoning abilities that generalize to harder test examples.
Effective for few-shot and data-efficient QA
Another strength of DRAGON is few-shot and data-efficient QA. For each QA dataset, we tried finetuning DRAGON and the baseline models with only 10% or 1% of the available training data. We find that DRAGON provides large improvements in these low-resource paradigms. This suggests that DRAGON internalized more knowledge thanks to the self-supervised pretraining with knowledge graphs.
We introduced DRAGON, a new method to pretrain a deeply fused foundation model from text and knowledge graphs (KG). Specifically, we design a bidirectional cross-modal model for text and KG, and train the model using two joint self-supervised tasks: language modeling and KG link prediction.
DRAGON can be used as a drop-in replacement for existing BERT models, and can be finetuned to solve various NLP tasks. In particular, DRAGON achieves significant performance improvements for knowledge- and reasoning-intensive applications, such as complex question answering that involves commonsense/biomedical knowledge and multi-step reasoning.
The pretrained DRAGON models are made publicly available. We hope that they could be helpful for your projects and research. Finally, we think that DRAGON opens up many exciting future projects, such as generalizing it to GPT or sequence-to-sequence style language models to perform knowledge-grounded text generation.
This blog post is based on the paper:
The models, code and data are available on GitHub. If you have questions, please feel free to email us.
Many thanks to my collaborators and advisors, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang and Jure Leskovec for their help, and to the members of the Stanford SNAP group, P-Lambda group, and AI lab for their valuable feedback.
For more information, please visit Electronic Hearing Protector Supply, Custom Shooting Hearing Protection, Bluetooth Headset for Construction Workers.