大规模、详细、多样化的多轮聊天数据（以及模型）
We report three evaluations in this section: Alpaca-Eval from Stanford, Evol-instruct from Microsoft's WizardLM, and our curated evaluation set. Evaluations of modern LLMs may be biased and affected by many factors, we are also actively working on more comprehensive evaluation methods. #### Alpaca-Eval [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/) is a leaderboard specifically designed for evaluating LLMs. The leaderboard is created based on the win-rate against Text-Davince-003 automatically evaluated by GPT-4. #### Evol-instruct This dataset is constructed with an evolutionary strategy by rewriting the instructions through multiple rounds to obtain instructions at different complexity levels. The benchmark is developed by the [WizardLM project](https://github.com/nlpxucan/WizardLM), another excellent chat language model! #### Our Evaluation Set We curate an evaluation set, encompassing the [Vicuna Benchmark](https://lmsys.org/blog/2023-03-30-vicuna/) and additional 300 questions and instructions generated by GPT-4. The questions/instructions cover a wide range of topics, including commonsense, world knowledge, professional knowledge (specifically physics and biology) , mathematics, and writing tasks on different levels of difficulty. We use GPT-4 for evaluation. [Here](data/ultra_eval.json) is the dataset.
Examples of UltraLM- Example A: Question of Common Knowledge > **User:** Which animal is known as man's best friend? > **UltraLM:** There are many animals that humans enjoy as companion animals, but one of the absolute standout answers to the question "which animal is known as a man's best friend?" is the domestically-bred dog. Domesticated dogs have been by humans' sides for tens of thousands of years, providing faithful companionship and help; guarding people and property; and lending emotional support in everything from long, lonely days to military deployments. Dogs come in an almost mind-boggling variety of breeds of various sizes, with different energy levels, personalities, and temperaments, meaning there's almost certainly a pup out there who will be a perfect fit for whoever needs it most. - Example B: Commonsense Reasoning > **User:** Which weighs more: a pound of feathers or a pound of bricks? > **UltraLM:** A pound of feathers and a pound of bricks both weigh exactly one pound. This is because one pound is a unit of weight, just as a foot is a unit of length or a second is a unit of time. We use the pound (or other unit of weight) to describe the weight of different objects, regardless of what that object is made of. So the answer to your question is that a pound of feathers and a pound of bricks both weigh the same amount. - Example C: Reasoning > **User:** If a farmer has 17 sheep and all but 9 die, how many are left? > **UltraLM:** The total number of sheep that are left is 9. - Example D: Reasoning > **User:** If a red house is made from red bricks, and a blue house is made from blue bricks, what is a greenhouse made from? > **UltraLM:** A greenhouse is not made from green bricks, but rather from materials such as glass or plastic that allow light to pass through and create a controlled environment for plants to grow. Greenhouses can come in a variety of colors, including green, but the color of the building itself does not affect the plants growing inside. - Example E: Creation > **User:** Write a poem about electromagnetism. > **UltraLM:** Electromagnetism is a force we can't ignore, It's the reason for many things we adore. In a magnetic field, a wire will glow, And the electrons will align, both old and new. Electricity and magnetism are hand in hand, One creates a field, the other has a brand. The magnetic field can push and pull, On a wire with current, it will react in full. The attraction and repulsion it brings, Are the reason for many things. From motors to generators, and much more in between, Electromagnetism is at the root of what it means. It's a force that's all around us, From the compass to the light switch. With its power, it can do much, From a simple needle to a rocket's thrust. It's a force we can't deny, For it's a part of our lives. Electromagnetism is something to admire, For its strength and its might, it has no equal, it's sure. - Example F: Creation > **User:** Write a short story about the first second of the universe. > **UltraLM:** In the first second of the universe, darkness reigned supreme. There was no light, no sound, no shape, no substance. Nothing yet was, but there was an intense power, a raw energy waiting to be unleashed. Within infinity, the first second held the potential for creation in its grasp. And then, with a mighty bang, a bright point of light emerged into existence. In it, the most fundamental particles were born, their forms yet unknown as they frenetically collided in a chaotic dance of creation. Within this primordial light, every force of nature and every component of life was stirred from pure energy to physical reality. The first atoms quickly assembled, and from these, stars and galaxies began to form. Their fusion creating a hot environment that pushed back the very void, shaping the cosmos through expanding and cooling waves. In this cosmic first, no structure of experience, not even time or space, could establish its rule. For an instant, the entirety of the universe existed in an embryonic nexus, a cosmic singularity from which all future existence originated. It was a microcosm forging the macrocosm of which it ultimately becomes a part, its moment of birth a metaphor for the cycles of creation that permeate the multiverse(s).From that single point of light, an infinite universe unfolded. For centuries, eons ahead, and forevermore, it remained the most foundational blink of its unceasing journey into the depths of the Great Beyond. The first second is the beginning of everything.
## Overview of UltraChat This project aims to construct *open-source, large-scale, and multi-round* dialogue data powered by Turbo APIs to facilitate the construction of powerful language models with general conversational capability. In consideration of factors such as safeguarding privacy, **we do not directly use any data available on the Internet as prompts**. UltraChat is composed of three sectors- **Questions about the World**: The dialogue data in this sector is derived from a wide range of inquiries related to concepts, entities, and objects from the real world. The topics covered are extensive, spanning areas such as technology, art, and entrepreneurship. - ✍ **Writing and Creation**: The dialogue data in this sector is driven by the demands for writing/creation from scratch, and encompasses any tasks that an AI assistant may aid within the creative process, spanning from email composition to crafting narratives and plays, and beyond. - **Assistance on Existent Materials**: The dialogue data in this sector is generated based on existing materials, including but not limited to rewriting, continuation, summarization, and inference, covering a diverse range of topics.
*Disclaimer: Although the process of building UltraChat does NOT involve any publicly available benchmark data, scaling to a certain extent may still result in some overlap in some evaluation benchmarks. We would like to emphasize again that **all the data is automatically generated (including the instructions and responses)**, and we do not insert any open benchmark data. For example, UltraChat was released (April, 2023) **earlier** than Alpaca Eval (May, 2023). We encourage users to closely monitor such phenomena, while we are also actively considering how to evaluate LLMs more properly.* An Example of UltraChat## Data The dataset is intended solely for research and educational purposes and should not be construed as reflecting the opinions or views of the creators, owners, or contributors of this dataset. And it is distributed under the MIT license. ### Data Release [Explore](http://39.101.77.220/) the data before downloading, or use [Atlas explorer](https://atlas.nomic.ai/map/0ce65783-c3a9-40b5-895d-384933f50081/a7b46301-022f-45d8-bbf4-98107eabdbac). - [Huggingface Datasets Host](https://huggingface.co/datasets/stingning/ultrachat
暂无开放 Issues，或尚未同步最近议题。