云中的嵌入
#showdev #python #machinelearning #nlp

本文是txtai的教程系列的一部分,txtai,一个AI驱动的语义搜索平台。

嵌入是提供语义搜索的引擎。数据被转换为嵌入矢量,其中类似的概念会产生相似的向量。这些向量都构建了大小的索引。索引用于查找具有相同含义的结果,不一定是相同的关键字。

除了本地存储外,还可以将嵌入与云存储同步。鉴于TXTAI是一种完全封装的索引格式,因此Cloud Sync只是将一组文件转移到云存储的问题。这可以是对象存储,例如AWS S3/Azure Blob/Google Cloud或Hugging Face Hub。有关可用选项的更多详细信息,请参见documentation。还有一个可用的article,涵盖了如何在云对象存储中构建和存储索引。

本文将介绍一个从拥抱面枢纽加载嵌入索引的示例。

安装依赖项

安装txtai和所有依赖项。

# Install txtai
pip install txtai

与拥抱式枢纽集成

拥抱面轮有各种各样的型号,数据集和示例应用程序,可用于开始您的项目。现在,这包括txtai索引ð¥ð¥ð¥

让我们加载标准Introducing txtai示例中使用的嵌入。

from txtai.embeddings import Embeddings

# Load the index from the Hub
embeddings = Embeddings()
embeddings.load(provider="huggingface-hub", container="neuml/txtai-intro")

注意两个字段,providercontainerprovider领域告诉TXTAI在拥抱的脸上枢纽中寻找索引。 container字段设置了目标存储库。

print("%-20s %s" % ("Query", "Best Match"))
print("-" * 50)

# Run an embeddings search for each query
for query in ("feel good story", "climate change", "public health story", "war", "wildlife", "asia", "lucky", "dishonest junk"):
    # Get to the top result
    result = embeddings.search(query, 1)[0]

    # Print text
    print("%-20s %s" % (query, result["text"]))
Query                Best Match
--------------------------------------------------
feel good story      Maine man wins $1M from $25 lottery ticket
climate change       Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg
public health story  US tops 5 million confirmed virus cases
war                  Beijing mobilises invasion craft along coast as Taiwan tensions escalate
wildlife             The National Park Service warns against sacrificing slower friends in a bear attack
asia                 Beijing mobilises invasion craft along coast as Taiwan tensions escalate
lucky                Maine man wins $1M from $25 lottery ticket
dishonest junk       Make huge profits without work, earn up to $100,000 a day

如果您以前看过Txtai,这是一个经典的例子。不过,最大的区别是,该索引是从拥抱的脸部集线器上加载的,而不是动态构建。

Wikipedia搜索与Txtai

让我们尝试使用Wikipedia index available on the Hugging Face Hub
更有趣的东西

from txtai.embeddings import Embeddings

# Load the index from the Hub
embeddings = Embeddings()
embeddings.load(provider="huggingface-hub", container="neuml/txtai-wikipedia")

现在运行一系列搜索以显示此索引中可用的数据类型。

import json

for x in embeddings.search("Roman Empire", 1):
  print(json.dumps(x, indent=2))
{
  "id": "Roman Empire",
  "text": "The Roman Empire ( ; ) was the post-Republican period of ancient Rome. 
As a polity, it included large territorial holdings around the Mediterranean Sea in Europe, North Africa, and Western Asia, and was ruled by emperors. From the accession of Caesar Augustus as the first Roman emperor to the military anarchy of the 3rd century, it was a Principate with Italia as the metropole of its provinces and the city of Rome as its sole capital. The Empire was later ruled by multiple emperors who shared control over the Western Roman Empire and the Eastern Roman Empire. The city of Rome remained the nominal capital of both parts until AD 476 when the imperial insignia were sent to Constantinople following the capture of the Western capital of Ravenna by the Germanic barbarians. The adoption of Christianity as the state church of the Roman Empire in AD 380 and the fall of the Western Roman Empire to Germanic kings conventionally marks the end of classical antiquity and the beginning of the Middle Ages. Because of these events, along with the gradual Hellenization of the Eastern Roman Empire, historians distinguish the medieval Roman Empire that remained in the Eastern provinces as the Byzantine Empire.",
  "score": 0.8913329243659973
}
for x in embeddings.search("How does a car engine work", 1):
  print(json.dumps(x, indent=2))
{
  "id": "Internal combustion engine",
  "text": "An internal combustion engine (ICE or IC engine) is a heat engine in which the combustion of a fuel occurs with an oxidizer (usually air) in a combustion chamber that is an integral part of the working fluid flow circuit. In an internal combustion engine, the expansion of the high-temperature and high-pressure gases produced by combustion applies direct force to some component of the engine. The force is typically applied to pistons (piston engine), turbine blades (gas turbine), a rotor (Wankel engine), or a nozzle (jet engine). This force moves the component over a distance, transforming chemical energy into kinetic energy which is used to propel, move or power whatever the engine is attached to. This replaced the external combustion engine for applications where the weight or size of an engine was more important.",
  "score": 0.8664469122886658
}
for x in embeddings.search("Who won the World Series in 2022?", 1):
  print(json.dumps(x, indent=2))
{
  "id": "2022 World Series",
  "text": "The 2022 World Series was the championship series of Major League Baseball's (MLB) 2022 season. The 118th edition of the World Series, it was a best-of-seven playoff between the American League (AL) champion Houston Astros and the National League (NL) champion Philadelphia Phillies. The Astros defeated the Phillies in six games to earn their second championship. The series was broadcast in the United States on Fox television and ESPN Radio. ",
  "score": 0.8889098167419434
}
for x in embeddings.search("What was New York called under the Dutch?", 1):
  print(json.dumps(x, indent=2))
{
  "id": "Dutch Americans in New York City",
  "text": "Dutch people have had a continuous presence in New York City for nearly 400 years, being the earliest European settlers. New York City traces its origins to a trading post founded on the southern tip of Manhattan Island by Dutch colonists in 1624. The settlement was named New Amsterdam in 1626 and was chartered as a city in 1653. Because of the history of Dutch colonization, Dutch culture, politics, law, architecture, and language played a formative role in shaping the culture of the city. The Dutch were the majority in New York City until the early 1700s and the Dutch language was commonly spoken until the mid to late-1700s. Many places and institutions in New York City still bear a colonial Dutch toponymy, including Brooklyn (Breukelen), Harlem (Haarlem), Wall Street (Waal Straat), The Bowery (bouwerij (\u201cfarm\u201d), and Coney Island (conyne).",
  "score": 0.8840358853340149
}

现在可能很清楚这些结果如何与另一个组件(例如LLM提示)结合起来,以构建基于对话QA的系统!

通过普及过滤

让我们尝试最后一个查询。这是一个通用查询,只有很多相似性搜索的匹配结果。

for x in embeddings.search("Boston", 1):
  print(json.dumps(x, indent=2))
{
  "id": "Boston (song)",
  "text": "\"Boston\" is a song by American rock band Augustana, from their debut album All the Stars and Boulevards (2005). It was originally produced in 2003 by Jon King for their demo, Midwest Skies and Sleepless Mondays, and was later re-recorded with producer Brendan O'Brien for All the Stars and Boulevards.",
  "score": 0.8729256987571716
}

虽然结果是关于波士顿的,但这并不是最受欢迎的结果。这是百分位数领域来提供帮助的地方。可以根据页面视图的数量过滤结果。

for x in embeddings.search("SELECT id, text, score, percentile FROM txtai WHERE similar('Boston') AND percentile >= 0.99", 1):
  print(json.dumps(x, indent=2))
{
  "id": "Boston",
  "text": "Boston, officially the City of Boston, is the state capital and most populous city of the Commonwealth of Massachusetts, as well as the cultural and financial center of the New England region of the United States. It is the 24th-most populous city in the country. The city boundaries encompass an area of about  and a population of 675,647 as of 2020. It is the seat of Suffolk County (although the county government was disbanded on July 1, 1999). The city is the economic and cultural anchor of a substantially larger metropolitan area known as Greater Boston, a metropolitan statistical area (MSA) home to a census-estimated 4.8\u00a0million people in 2016 and ranking as the tenth-largest MSA in the country. A broader combined statistical area (CSA), generally corresponding to the commuting area and including Providence, Rhode Island, is home to approximately 8.2\u00a0million people, making it the sixth most populous in the United States.",
  "score": 0.8668985366821289,
  "percentile": 0.9999025135905505
}

此查询添加了一个额外的过滤器,仅匹配访问的Wikipedia页面的顶部1%的结果。

包起来

本文介绍了如何从云存储中加载嵌入式索引。拥抱面线是共享模型,数据集,示例应用程序和现在的TXTAI嵌入索引的绝佳资源。当索引时间长或需要大量的GPU资源时,这特别有用。

期待看到未来几个月中社区分享的嵌入方式!