Semantic search with Pinecone and OpenAI

Mar 24, 2023
Open in Github

In this guide you will learn how to use the OpenAI Embedding API to generate language embeddings, and then index those embeddings in the Pinecone vector database for fast and scalable vector search.

This is a powerful and common combination for building semantic search, question-answering, threat-detection, and other applications that rely on NLP and search over a large corpus of text data.

The basic workflow looks like this:

Embed and index

  • Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data).
  • Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies.

Search

  • Pass your query text or document through the OpenAI Embedding API again.
  • Take the resulting vector embedding and send it as a query to Pinecone.
  • Get back semantically similar documents, even if they don't share any keywords with the query.

Architecture overview

Let's get started...

We first need to setup our environment and retrieve API keys for OpenAI and Pinecone. Let's start with our environment, we need HuggingFace Datasets for our data, and the OpenAI and Pinecone clients:

!pip install -qU \
    pinecone-client==3.0.2 \
    openai==1.10.0 \
    datasets==2.16.1
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.4/201.4 kB 2.0 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 225.1/225.1 kB 12.1 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 507.1/507.1 kB 12.4 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 4.4 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 9.7 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 7.6 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.9/76.9 kB 4.7 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 3.8 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 5.2 MB/s eta 0:00:00
[?25hERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
llmx 0.0.15a0 requires cohere, which is not installed.
llmx 0.0.15a0 requires tiktoken, which is not installed.
tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.

from openai import OpenAI

client = OpenAI(
    api_key="OPENAI_API_KEY"
)  # get API key from platform.openai.com

We can now create embeddings with the OpenAI Ada similarity model like so:

MODEL = "text-embedding-3-small"

res = client.embeddings.create(
    input=[
        "Sample document text goes here",
        "there will be several phrases in each batch"
    ], model=MODEL
)
res
CreateEmbeddingResponse(data=[Embedding(embedding=[-0.0007019874756224453, 0.017813093960285187, 0.028484342619776726, -0.01655358262360096, -0.04467806592583656, -0.03371616080403328, 0.02429058402776718, -0.015460160560905933, 0.0147542804479599, -0.006034583784639835, 0.03413138538599014, -0.010325227864086628, 0.004678186494857073, -0.006508630700409412, 0.06505170464515686, 0.07252573221921921, -0.004359848331660032, 0.013335599564015865, -0.026740403845906258, 0.03684417903423309, 0.004705868195742369, 0.04675418511033058, -0.021384017542004585, 0.03440820053219795, 0.012664321810007095, -0.021896127611398697, -0.040553510189056396, -0.024913419038057327, 0.05112787336111069, -0.06244963780045509, -0.01853281632065773, -0.04124554991722107, 0.01914181001484394, -0.03044973500072956, -0.040165968239307404, 0.054698795080184937, 0.06560533493757248, 0.0040449704974889755, -0.049799155443906784, -0.04819362610578537, -0.009563985280692577, -0.012491311877965927, 0.0549202486872673, 0.009536303579807281, -0.0008953260257840157, 0.03521096706390381, 0.008242189884185791, -0.019653920084238052, -0.005422128830105066, 0.056221283972263336, 0.005581297911703587, -0.020747341215610504, -0.0012690272415056825, 0.022989550605416298, 0.03498951345682144, -0.002600338077172637, -0.024802692234516144, 0.02240823581814766, -0.00995152723044157, -0.02694801613688469, 0.009224885143339634, 0.017453234642744064, -0.01533559337258339, 0.033273257315158844, -0.015266389586031437, 0.03438051789999008, -0.045314740389585495, 0.01100342720746994, -0.020567411556839943, 0.0014826945262029767, 0.057300865650177, 0.017259463667869568, -0.004224900621920824, 0.017688527703285217, 0.010048412717878819, -0.00761243375018239, -0.05719013884663582, -0.005692024249583483, -0.031695406883955, -0.019764646887779236, 0.030505098402500153, 0.03241512551903725, -0.0039446246810257435, -0.04224208742380142, -0.0654946118593216, 0.01167470496147871, -0.06571606546640396, 0.04395834356546402, -0.04747390374541283, -0.02152242697775364, -0.01990305446088314, 0.004453273490071297, 0.00022145261755213141, -0.0035570827312767506, 0.02429058402776718, 0.005408287979662418, -0.062172822654247284, 0.008103781379759312, -0.020802704617381096, -0.008539766073226929, 0.06084410473704338, -0.023418614640831947, -0.02286498248577118, -0.011134914122521877, 0.015515523962676525, 0.01594458892941475, 0.028152164071798325, -0.021231770515441895, -0.0457853302359581, -0.02380615659058094, -0.06997902691364288, -0.0011297543533146381, 0.019127970561385155, 0.03828362002968788, -0.009114159271121025, -0.005788909737020731, 0.04296180605888367, -0.036899540573358536, 0.008539766073226929, -0.024027608335018158, -0.03471269831061363, -0.015709295868873596, -0.005726626142859459, -0.03623518347740173, -0.009197204373776913, -0.05115555599331856, 0.021868444979190826, -0.026422064751386642, -0.03244280815124512, 0.025591617450118065, 0.0505465604364872, -0.04182686284184456, 0.017826935276389122, -0.03828362002968788, -0.028332093730568886, 0.015543205663561821, -0.04005524143576622, 0.05281645059585571, -0.028456661850214005, 0.037176359444856644, 0.014574350789189339, -0.0366504080593586, 0.02651895023882389, 0.03128018230199814, -0.020041462033987045, -0.0295085608959198, -0.024152176454663277, -0.01840824820101261, -0.02697569690644741, 0.01929405890405178, -0.02413833513855934, 0.04495488107204437, -0.04628359526395798, 0.021896127611398697, 0.03634591028094292, -0.023584702983498573, 0.04617287218570709, -0.020941112190485, -0.039556972682476044, 0.023238683119416237, -0.012885774485766888, -0.0464496873319149, 0.012110689654946327, 0.017840776592493057, -0.016387494280934334, -0.02390304207801819, 0.014948051422834396, -0.014242171309888363, -0.06770914047956467, 0.012027645483613014, -0.01006225310266018, -0.04456733912229538, -0.012345983646810055, -0.024498196318745613, -0.021591629832983017, 0.03573691472411156, -0.013300998136401176, -0.017397871240973473, 0.020235233008861542, 0.03252585232257843, 0.010871939361095428, 0.04827667027711868, 0.012449788860976696, 0.02195149101316929, 0.024221379309892654, -0.0514046885073185, 0.010235263034701347, 0.047086361795663834, 0.02377847395837307, -0.011882317252457142, 0.003674729261547327, 0.029287109151482582, -0.02719714865088463, -0.03260889649391174, -0.021148724481463432, -0.006574374623596668, 0.06078874319791794, -0.0018460152205079794, 0.009038034826517105, 0.03593068569898605, 0.00554669601842761, 0.05655346065759659, -0.05544620007276535, 0.007384060882031918, -0.06632506102323532, -0.023709271103143692, -0.03305180370807648, -0.010657407343387604, -0.017577800899744034, -0.06604824215173721, 0.047750718891620636, 0.009432497434318066, 0.02484421618282795, 0.0011963631259277463, -0.03487878665328026, -0.02575770765542984, -0.009847721084952354, 0.04841507971286774, 0.028428979218006134, -0.003920403309166431, -0.010581282898783684, 0.03554314374923706, 0.06654651463031769, 0.03548778221011162, 0.011951521039009094, -0.008574368432164192, -0.01248439121991396, -0.017563961446285248, -0.017909979447722435, -0.0011816573096439242, 0.017550120130181313, 0.002157432958483696, 0.02314179763197899, -0.03496183082461357, 0.02264353074133396, -0.005792370066046715, -0.008048418909311295, 0.003657428314909339, -0.015736976638436317, 0.007937692105770111, -0.06925930827856064, 0.008865024894475937, 0.031169455498456955, 0.012671241536736488, 0.030809596180915833, -0.031473953276872635, -0.041162505745887756, -0.07523852586746216, 0.032664261758327484, 0.05024206265807152, 0.02274041622877121, 0.012006884440779686, 0.016816558316349983, -0.008435960859060287, -0.0019152191234752536, 0.0034999893978238106, -0.006055344827473164, 0.02200685441493988, 0.01652590185403824, -0.004269883036613464, -0.05325935408473015, 0.002250858349725604, -0.023736951872706413, 0.02142554149031639, -0.008256030268967152, -0.0020121047273278236, 0.022366713732481003, 0.005346004385501146, 0.010512079112231731, -0.03396529331803322, -0.008781980723142624, 0.051044829189777374, 0.04420747980475426, -0.0197508055716753, -0.05840812623500824, 0.03313484787940979, 0.0055328551679849625, -0.023307887837290764, 0.020954953506588936, 0.007079563569277525, -0.025646980851888657, 0.003546701977029443, -0.02420753985643387, 0.000534167920704931, -0.019363263621926308, -0.01060896459966898, 0.044705748558044434, 0.01579234004020691, 0.0025484352372586727, 0.007785443682223558, 0.033909931778907776, -0.03496183082461357, 0.014408260583877563, -0.03833898529410362, -0.02743244357407093, 0.016262926161289215, -0.012159132398664951, 0.022394396364688873, 0.04127323254942894, 0.010539760813117027, 0.030837276950478554, 0.026629677042365074, 0.006875411607325077, 0.02231135033071041, 0.017300985753536224, 0.024608921259641647, 0.017176419496536255, -0.028428979218006134, 0.033937614411115646, -0.02316948026418686, 0.03283035010099411, -0.033273257315158844, -0.031446270644664764, -0.04728013277053833, -0.032359763979911804, -0.024193698540329933, 0.08221428096294403, 0.008961910381913185, -0.041411638259887695, -0.04027669504284859, 0.021079521626234055, -0.03795144334435463, 0.058242037892341614, -0.0028321712743490934, 0.012795808725059032, 0.03249816969037056, 0.015114140696823597, 0.01883731409907341, 0.016664309427142143, -0.04525937885046005, -0.0077716028317809105, -0.04343239590525627, 0.005761228036135435, -0.026726562529802322, 0.027086423709988594, -0.008484403602778912, 0.004775071982294321, 0.05544620007276535, 0.021051838994026184, 0.02923174574971199, -0.05719013884663582, -0.007397901266813278, 0.015183345414698124, 0.01507261861115694, 0.003287187311798334, -0.014948051422834396, -0.007889249362051487, 0.027695417404174805, 0.029093338176608086, 0.014532827772200108, -0.023307887837290764, 0.03695490583777428, 0.035072557628154755, 0.021010316908359528, 0.04326630383729935, -0.0334116630256176, 0.0013771584490314126, -0.023183321580290794, 0.01615219935774803, -0.018117591738700867, -0.02304491214454174, 0.0030034510418772697, 0.06455343961715698, -0.006110708229243755, -0.0051972162909805775, 0.025439368560910225, 0.0034550067503005266, -0.03103104792535305, 0.05458806827664375, 0.004903099499642849, 0.0028217907529324293, -0.01820063777267933, -0.008512085303664207, -0.01135636679828167, -0.06261572986841202, -0.013356360606849194, -0.03197222203016281, 0.020927272737026215, 0.01716257818043232, -0.011944600380957127, -0.048691894859075546, 0.028456661850214005, -0.035404738038778305, -0.012131450697779655, 0.04669881984591484, 0.013314838521182537, -0.021660834550857544, 0.04066423699259758, -0.006176451686769724, -0.00014338191249407828, 0.007051881868392229, 0.017633164301514626, 0.024124493822455406, 0.03922479599714279, -0.0070311203598976135, 0.029397834092378616, -0.010491318069398403, 0.0386434830725193, 0.0016418634913861752, -0.04393066465854645, 0.06665723770856857, -0.0022162562236189842, 0.006643578410148621, -0.034159064292907715, -0.048055216670036316, 0.004778532311320305, -0.03382688760757446, -0.0206089336425066, 0.011197198182344437, -0.05658114328980446, 0.05212441086769104, -0.021300973370671272, 0.014657394960522652, 0.003185111563652754, -0.00039078600821085274, 0.019930735230445862, 0.011806192807853222, 0.061231646686792374, 0.018892675638198853, -0.012491311877965927, -0.008906546980142593, -0.057079412043094635, -0.012989579699933529, -0.025965319946408272, -0.018311362713575363, 0.04271267354488373, 0.031141774728894234, -0.06964685022830963, 0.011557058431208134, -0.0566365085542202, -0.016927285119891167, -0.0031366688199341297, 0.027930710464715958, -0.006508630700409412, -0.018518975004553795, 0.015418638475239277, -0.0327473059296608, -0.007522468455135822, -0.019986098632216454, -0.006335620768368244, -0.005356385372579098, 9.104643686441705e-05, -0.0473078154027462, -0.007730080280452967, -0.03994451463222504, -0.00055146892555058, 0.02548089250922203, 0.005280260927975178, 0.002705874154344201, -0.025356324389576912, 0.01835288479924202, -0.024678125977516174, 0.009792357683181763, -0.03546009957790375, 0.04575764760375023, 0.021688515320420265, 0.039252474904060364, 0.0332178920507431, 0.014311375096440315, -0.017148736864328384, 0.005525934975594282, 0.013833868317306042, -0.013003421016037464, 0.024249061942100525, -0.04213136062026024, -0.044152114540338516, 0.015709295868873596, 0.01685808040201664, 0.024899577721953392, -0.006923854351043701, -0.034214429557323456, -0.01743939332664013, 0.022325191646814346, 0.046477366238832474, -0.014110683463513851, -0.05702405050396919, 0.011411730200052261, -0.02453971840441227, 0.010553601197898388, 0.00746018486097455, 0.025384007021784782, -0.00020728743402287364, -0.045674603432416916, 0.05912784859538078, 0.0036124459002166986, 0.09826959669589996, -0.007799284532666206, 0.01802070625126362, 0.019916893914341927, -0.015307911671698093, -0.0057473876513540745, 0.018186796456575394, 0.029674651101231575, -0.015903066843748093, -0.01807606965303421, -0.04487183690071106, 0.0194186270236969, -0.04498256370425224, 0.020858068019151688, -0.004321786109358072, 0.02139785885810852, -0.005131472367793322, -0.012110689654946327, 0.020622774958610535, 0.04387529939413071, -0.009266408160328865, -0.057633042335510254, 0.009619347751140594, 0.011217959225177765, 0.034214429557323456, -0.02661583572626114, 0.027667736634612083, 0.010172979906201363, 0.004107254091650248, -0.002717984840273857, 0.03277498856186867, 0.02112104371190071, 0.026408225297927856, -0.0239860862493515, -0.007494787219911814, -0.002707604318857193, 0.0024273283779621124, 0.012235256843268871, -0.016235245391726494, -0.003214523196220398, -0.025287121534347534, -0.0018702365923672915, -0.017024170607328415, 0.015127982012927532, 0.015432478860020638, 0.0075570703484117985, -0.038117531687021255, 0.03227671980857849, -0.004903099499642849, 0.014477464370429516, -0.04448429495096207, 0.003972306381911039, 0.032996438443660736, 0.03759158030152321, -0.009134920313954353, 0.006996518466621637, 0.004325246438384056, 0.018768109381198883, -0.021605471149086952, -0.020775023847818375, 0.006114168558269739, 0.024650445207953453, 0.014256012625992298, -0.013148749247193336, 0.010235263034701347, 0.009003433398902416, 0.011072630994021893, 0.02603452280163765, 0.001029408653266728, -0.007224891800433397, 0.015418638475239277, 0.0066712601110339165, 0.00932869128882885, 0.019653920084238052, -0.0012301000533625484, 0.0024671205319464207, -0.038671161979436874, 0.0008615890983492136, -0.0013347710482776165, -0.025231758132576942, 0.04966074973344803, -0.0009558794554322958, 0.004404830746352673, 0.07047729194164276, -0.0011868475703522563, -0.014089922420680523, 0.014117604121565819, -0.009044955484569073, -0.021259451285004616, 0.014657394960522652, -0.022823460400104523, 0.0032283638138324022, -0.019958417862653732, 0.007432503625750542, 0.012435948476195335, 0.012186814099550247, 0.0028979151975363493, 7.05015190760605e-05, 0.0034792281221598387, -0.03141859173774719, 0.020885750651359558, 0.03186149522662163, 0.0025674663484096527, 0.0023996466770768166, 0.018574338406324387, 0.02017986960709095, -0.021287132054567337, -0.02012450620532036, 0.03526632860302925, 0.011342526413500309, -0.01850513368844986, 0.0017162577714771032, -0.00020285406208131462, -0.018602019175887108, -0.0024723107926547527, 0.018546655774116516, -0.023557022213935852, 0.00425258232280612, -0.02362622693181038, -0.0009489590884186327, 0.011515536345541477, -0.021868444979190826, -0.00554669601842761, -0.0008101186831481755, -0.0412178672850132, -0.019515512511134148, 0.026117568835616112, -0.01460203155875206, -0.022602006793022156, -0.007792363874614239, -0.003243934828788042, -0.027626214548945427, 0.029674651101231575, 0.015446320176124573, -0.008387518115341663, 0.0034359758719801903, -0.014269853010773659, 0.02380615659058094, 0.014117604121565819, -0.016567423939704895, 0.015169504098594189, -0.0040934132412076, -0.010186820290982723, -0.05713477358222008, 0.0025224837008863688, 0.011785431765019894, 0.01749475672841072, 0.04667113721370697, -0.0183252040296793, 0.002811410231515765, -0.02337709255516529, 0.04514865204691887, -0.011875396594405174, -0.016636628657579422, -0.01642901636660099, 0.0030743852257728577, -0.012830411083996296, 0.042601946741342545, -0.03714867681264877, 0.0457853302359581, -0.0005817456403747201, 0.01199304312467575, -0.000565309717785567, 0.02426290139555931, -0.013079545460641384, -0.0009498241124674678, 0.011342526413500309, 0.001628022757358849, 0.02225598879158497, 0.025536254048347473, 0.023875359445810318, -0.026435906067490578, -0.029757695272564888, -0.023363251239061356, -0.021204087883234024, -0.018186796456575394, 0.04763999581336975, -0.02240823581814766, -0.013280236162245274, 0.0024619302712380886, 0.016664309427142143, -0.002160893054679036, -0.003823517821729183, -0.012013804167509079, 0.008643572218716145, -0.05051887780427933, 0.007342538330703974, 0.041134823113679886, -0.001652244129218161, 0.016055313870310783, 0.003986147232353687, -0.012304460629820824, -0.03197222203016281, 0.02368158847093582, 0.0035017195623368025, 0.013162589631974697, -0.0015095110284164548, -0.010830417275428772, -0.035127922892570496, -0.0008516410016454756, -0.009840800426900387, 0.0420759953558445, 0.0034636573400348425, 0.010747372172772884, 0.02157778851687908, -0.008532846346497536, -0.022020693868398666, -0.004920400213450193, -0.00054195336997509, 0.02707258239388466, 0.012311381287872791, -0.01199304312467575, -0.0033667718525975943, -0.013300998136401176, -0.039252474904060364, -0.04033205658197403, 0.005179915111511946, -0.020553570240736008, 0.039612337946891785, -0.007563991006463766, -0.055861420929431915, 0.007868488319218159, 0.011854635551571846, 0.05002060905098915, -0.051210917532444, 0.03460197150707245, -0.008131463080644608, -0.008366757072508335, 0.00452593807131052, 0.006851190235465765, -0.02225598879158497, -0.020871909335255623, 0.001575254718773067, -0.015031096525490284, -0.04761231318116188, 0.0039307838305830956, 0.019681600853800774, 0.012096849270164967, -0.026076044887304306, -0.016954965889453888, -0.017882298678159714, 0.03806217014789581, -0.024041449651122093, -0.00044896057806909084, -0.012664321810007095, -0.018186796456575394, 0.025384007021784782, 0.007100324612110853, 0.027183309197425842, 0.007224891800433397, 0.02570234425365925, -0.053508490324020386, -0.028484342619776726, -0.01370238047093153, -0.021896127611398697, -0.03368847817182541, 0.000546711147762835, -0.013300998136401176, 0.05057424306869507, -0.00329929799772799, -0.006262956652790308, -0.007896170020103455, 0.008692014962434769, -0.06604824215173721, -0.031197138130664825, -0.042878761887550354, 0.01191691868007183, -0.031446270644664764, -0.012359824031591415, 0.03368847817182541, -0.026989538222551346, -0.014809643849730492, -0.004643584601581097, 0.05724550038576126, 0.0041453163139522076, 0.002681652782484889, 0.039280157536268234, 0.004283723887056112, 0.00737022003158927, -0.07640115171670914, -0.020830387249588966, -0.0022698892280459404, 0.03346702829003334, -0.011612421832978725, 0.02575770765542984, 0.00858128909021616, -0.027238672599196434, -0.015155663713812828, -0.023100275546312332, 0.041743818670511246, -0.013439405709505081, -0.05685795843601227, 0.03701026737689972, -0.0030726550612598658, 0.016484379768371582, 0.012186814099550247, -0.016816558316349983, -0.013480927795171738, 0.02017986960709095, -0.005422128830105066, 0.00208476884290576, 0.004408291075378656, -0.003999988082796335, -0.0016764655010774732, -0.04733549803495407, -0.02109336107969284, 0.0035224806051701307, 0.014228330925107002, 0.004847736097872257, 0.010989585891366005, 0.0076331947930157185, -0.030200600624084473, -0.08531462401151657, -0.003108987119048834, 0.007384060882031918, -0.028428979218006134, -0.04218672215938568, 0.004896178841590881, 0.04941161349415779, 0.03878188878297806, 0.04603446274995804, -0.041134823113679886, 0.007197210099548101, -0.00874737836420536, -0.00035358889726921916, 0.006418665871024132, -0.0008429905283264816, -0.013467087410390377, 0.024401310831308365, 0.020069142803549767, -0.022200625389814377, -0.025079509243369102, -0.03977842628955841, -0.011937679722905159, 0.011480933986604214, 0.0019134889589622617, -0.03343934565782547, -0.028512025251984596, -0.035072557628154755, 0.018518975004553795, 0.04692027345299721, -0.003020751988515258, 0.021923808380961418, -0.02240823581814766, -0.029121018946170807, 0.005525934975594282, 0.006764685269445181, -0.017840776592493057, -0.017674686387181282, 0.004003447946161032, 0.010899621061980724, 0.013010341674089432, 0.004283723887056112, -0.01691344380378723, -0.030366690829396248, 0.037480857223272324, 0.013418644666671753, 0.015432478860020638, 0.0006444617174565792, 0.015557046048343182, -0.022172942757606506, 0.01094114314764738, -0.0029567384626716375, -0.014795802533626556, 0.04907943680882454, -0.003243934828788042, -0.01652590185403824, 0.011958441697061062, -0.04149468243122101, -0.019986098632216454, -0.0028546627145260572, 0.04343239590525627, -0.02185460552573204, -0.02466428466141224, -0.027792302891612053, 0.00534946471452713, -0.0298407394438982, -0.014311375096440315, -0.03258121758699417, 0.014214489609003067, -0.02261584810912609, 0.014117604121565819, 0.023861519992351532, 0.045093290507793427, 0.034795742481946945, 0.0017326937522739172, 0.007563991006463766, -0.015861542895436287, 0.0017309635877609253, 0.016941124573349953, 0.026131408289074898, 0.020138347521424294, 0.03756390139460564, -0.0034013737458735704, -0.006820048671215773, 0.031473953276872635, 0.00022275019728112966, 0.0145881911739707, 0.03191685676574707, -0.03438051789999008, 0.0006288908189162612, 0.053176309913396835, 0.05068496614694595, 0.010858098976314068, 0.0064947898499667645, -0.02319716103374958, -0.059515390545129776, -0.03219367563724518, -0.00327680679038167, -0.026601996272802353, -0.015446320176124573, -0.012152212671935558, 0.02200685441493988, 0.01759164221584797, 0.014297534711658955, -0.005487872753292322, -0.01972312293946743, -0.0018633161671459675, 0.025356324389576912, 0.01737019047141075, -0.00925256684422493, -0.02225598879158497, 0.006577834952622652, 0.0020553572103381157, 0.012283699586987495, -0.0055190143175423145, -0.039916835725307465, 0.010574362240731716, 0.021909968927502632, 0.01428369339555502, 0.010823496617376804, -0.008678174577653408, -0.00014143556472845376, 0.04182686284184456, -0.008055338636040688, -0.04556387662887573, -0.05082337558269501, -0.041107140481472015, -0.036318227648735046, 0.013017261400818825, -0.0163736529648304, 0.0075570703484117985, 0.01749475672841072, 0.021259451285004616, 0.010311387479305267, -0.010179899632930756, -3.3277363399975e-05, -0.021716197952628136, 0.026325179263949394, 0.014013797976076603, 0.0023183319717645645, -0.019003402441740036, 0.012768127024173737, -0.00655361358076334, 0.0022889203391969204, 0.00916260201483965, 0.016442857682704926, 0.0019463609205558896, -0.008643572218716145, 0.020249074324965477, -0.015114140696823597, -0.02920406311750412, -0.01443594228476286, 0.013051863759756088, -0.030062193050980568, -0.05323167145252228, -0.023432454094290733, -0.016207562759518623, -0.01878195069730282, -0.03728708252310753, -0.004861576948314905, 0.03346702829003334, -0.004916940350085497, 0.018588179722428322, -0.01069892942905426, -0.022975709289312363, 0.021439380943775177, 0.00745326466858387, -0.022602006793022156, 0.013633176684379578, 0.006944615859538317, -0.009979208931326866, -0.006131469272077084, -0.0050830296240746975, 0.025716185569763184, 0.011847714893519878, -0.0034480865579098463, 0.025107190012931824, 0.04639432206749916, -0.035072557628154755, 0.0004718843847513199, 0.017702369019389153, -0.02069197967648506, -0.018463611602783203, -0.06543924659490585, 0.0034169447608292103, 0.03703795000910759, -0.016138359904289246, 0.000552766490727663, 0.028844203799962997, 0.014186807908117771, 0.043792255222797394, 0.016622787341475487, -0.0032283638138324022, 0.02015218883752823, -0.0062837181612849236, 0.01691344380378723, 0.009923845529556274, 0.04326630383729935, 0.03413138538599014, 0.016816558316349983, -0.0073217772878706455, -0.00893422868102789, 0.032636579126119614, 0.0020553572103381157, 0.0007755166734568775, 0.0055709173902869225, 0.030809596180915833, -0.0648302510380745, 0.010124537162482738, -0.01182695385068655, 0.020567411556839943, -0.02362622693181038, -0.01460203155875206, -0.028055278584361076, -0.0054809520952403545, 0.03609677776694298, -0.02157778851687908, -0.02456739917397499, 0.03728708252310753, -0.03305180370807648, 0.006837349385023117, 0.03219367563724518, -0.03944624587893486, -0.045037925243377686, -0.005778529215604067, -0.037508536130189896, -0.016415175050497055, -0.03349470719695091, -0.011051869951188564, 0.030892640352249146, -0.004809673875570297, -0.003989607095718384, 0.021868444979190826, 0.04157773032784462, 0.003243934828788042, 0.024221379309892654, 0.002731825690716505, -0.027044901624321938, -0.005429049488157034, 0.02786150760948658, 0.025799229741096497, -0.01330791786313057, 0.01645669713616371, -0.020262913778424263, 0.0034255951177328825, 0.011058789677917957, -0.005214517004787922, 0.035349372774362564, 0.003792376024648547, 0.007349458523094654, 0.02469196729362011, 0.01761932298541069, 0.009321770630776882, 0.020996475592255592, -0.03219367563724518, -0.00394116435199976, -0.01460203155875206, 0.00752938911318779, -0.019460149109363556, 0.0036678090691566467, 0.039003342390060425, -0.02743244357407093, 0.0030345928389579058, 0.015252549201250076, -0.03133554384112358, 0.003212793031707406, 0.006370223127305508, -0.024982623755931854, 0.0283182542771101, 0.011882317252457142, -0.029342472553253174, -0.019044924527406693, 0.048636529594659805, 0.06876103579998016, 0.016262926161289215, -0.016691990196704865, 0.011044949293136597, 0.021024158224463463, -0.033882249146699905, 0.04060887172818184, -0.007861567661166191, -0.02091343142092228, 0.033300936222076416, 0.004653965122997761, -0.020844226703047752, -0.011847714893519878, 0.016816558316349983, -0.03742549195885658, -0.00940481573343277, -0.031778451055288315, 0.013501688838005066, 0.019335580989718437, -0.011065710335969925, 0.006207593716681004, 0.017882298678159714, -0.034823425114154816, 0.0002716254675760865, -0.014892688021063805, 0.04827667027711868, 0.018823472782969475, 0.027003377676010132, -0.006650499068200588, 0.004698947537690401, 0.00795845314860344, -0.023335568606853485, -0.00987540278583765, 0.03260889649391174, 0.010034571401774883, -0.025051826611161232, -0.017231781035661697, -0.024941101670265198, 0.019252536818385124, -0.0054809520952403545, 0.021840764209628105, 0.008546686731278896, 0.00422144029289484, 0.0016297528054565191, -0.007224891800433397, 0.05367457866668701, -0.0017076072981581092, -0.03886493295431137, -0.03803448751568794, -0.015612409450113773, -0.004435972776263952, 0.056802596896886826, 0.024221379309892654, 0.030339008197188377, 0.009508621878921986, -0.02350165881216526, -0.014781962148845196, -0.005688563920557499, -0.013653937727212906, -0.0077716028317809105, 0.01386846974492073, 0.003166080452501774, 0.02408297173678875, 0.01199304312467575, -0.01199304312467575, 0.0017456694040447474, -0.009307930245995522, -0.01415912713855505, 0.0035847641993314028, -0.02621445432305336, -0.01597226969897747, 0.02188228629529476, 0.004539778456091881, 0.0017266384093090892, 0.0071695283986628056, -0.002711064415052533, -0.008463642559945583, 0.005435969680547714, 0.10031803697347641, 5.19029563292861e-05, -0.037536218762397766, -0.023861519992351532, 0.01566777192056179, 0.004141855984926224, 0.002756047062575817, 0.023944564163684845, 0.01409684307873249, 0.005954999476671219, 0.015709295868873596, 0.002662621671333909, 0.028484342619776726, 0.0011228339280933142, 0.0186573825776577, 0.0054809520952403545, -0.019501671195030212, -0.05331471562385559, -0.0038823410868644714, -0.012885774485766888, 0.020401323214173317, 0.021453222259879112, 0.015252549201250076, 0.009058795869350433, -0.033245574682950974, -0.02444283291697502, -0.02038748189806938, -0.010975745506584644, 0.019404785707592964, 0.02590995654463768, 0.01835288479924202, 0.007965373806655407, 0.025231758132576942, -0.0014930750476196408, -0.02953624352812767, -0.02377847395837307, -0.0018027627374976873, -0.0007932502194307745, -0.007951533421874046, -0.024733489379286766, -0.017245622351765633, 0.038089849054813385, 0.026283657178282738, 0.007211050949990749, -0.00024437642423436046, -0.014712758362293243, -0.013736982829868793, 0.01132176537066698, 0.01338404230773449, 0.00200864439830184, -0.022325191646814346, 0.0233217291533947, -0.009453258477151394, 0.014394420199096203, 0.0381728932261467, -0.022048376500606537, 0.0063217803835868835, -0.022020693868398666, 0.04210367798805237, 0.021743878722190857, 0.01959855668246746, -0.006705862004309893, 0.03368847817182541, -0.003647047793492675, 0.02164699323475361, 0.011619341559708118, -0.002742206212133169, 0.009127999655902386, -0.002690303372219205, 0.030366690829396248, 0.01990305446088314, 0.004688567016273737, -0.014297534711658955, 0.03186149522662163, 2.8492560886661522e-05, 0.002628019778057933, 0.006231815088540316, -0.008643572218716145, 0.00972315389662981, 0.009204124100506306, -0.00319722224958241, -0.019404785707592964, 0.02608988620340824, 0.0247473306953907, 0.020069142803549767, 0.0063183200545609, -0.007840806618332863, -0.03457428887486458, 0.024899577721953392, 0.008961910381913185, 0.002230097074061632, 0.01289269421249628, 0.03213831037282944, 0.014422101899981499, 0.00620413338765502, 0.008629731833934784, 0.009598586708307266, 0.008172986097633839, 0.0017335587181150913, -0.011377127841114998, 0.005664342548698187, 0.016595104709267616, 0.00024762036628089845, 0.014117604121565819, 0.003692030441015959, 0.01474044006317854, -0.02694801613688469, -0.01474044006317854, -0.04780608415603638, 0.027363238856196404, 0.01522486750036478, -0.012692003510892391, 0.01287885382771492, -0.007972294464707375, 0.0039446246810257435, -0.0008503434364683926, -0.021868444979190826, 0.012345983646810055, -0.017273304983973503, 0.013584733940660954, -0.010076094418764114, 0.021467063575983047, 0.02453971840441227, -0.04498256370425224, 0.024982623755931854, 0.00768855819478631, 0.01716257818043232, 0.019778486341238022, 0.0020363260991871357, -0.041162505745887756, 0.02761237323284149, 0.011363287456333637, 0.0009602046920917928, 0.016788875684142113, -0.004256042651832104, -0.010290626436471939, -0.0030259424820542336, -0.012567436322569847, -0.007979214191436768, -0.015127982012927532, -0.025965319946408272, -0.01932174153625965, 0.0035570827312767506, -0.02033211849629879, 0.019515512511134148, 0.015321752987802029, -0.028027595952153206, 0.001583905192092061, -0.008664333261549473, -0.006996518466621637, 0.0067473845556378365, 0.015557046048343182, 0.009183363057672977, -0.009038034826517105, -0.03623518347740173, 0.01612451858818531, -0.028982611373066902, -0.04271267354488373, 0.010705850087106228, -0.003602065145969391, 0.024927260354161263, -0.016235245391726494, -0.011958441697061062, -0.0035345912910997868, 0.040470466017723083, -0.051210917532444, -0.051653821021318436, 0.017605483531951904, -0.006605516187846661, -0.00041024963138625026, 0.003342550480738282, -0.004522477742284536, 0.002607258502393961, 0.014186807908117771, -0.04282340034842491, -0.008954989723861217, 0.01386846974492073, 0.043819937855005264, -0.023266365751624107, 0.027487806975841522, -0.0103044668212533, -0.034242112189531326, -0.0008494784124195576, -0.0004294969839975238, 0.008332154713571072, 0.030920321121811867, -0.00897575169801712, -0.022394396364688873, -0.030588142573833466, -0.013065704144537449, 0.008269871585071087, 0.009868482127785683, -0.0008386652916669846, 0.004754310939460993, 0.001065740711055696, 0.049854520708322525, 0.01792382076382637, -0.0340760201215744, -0.023390932008624077, 0.011972282081842422, -0.029397834092378616, -0.004730089567601681, 0.05353616923093796, -0.006885792128741741, 0.014851165935397148, -0.0013944593956694007, -0.0008386652916669846, -0.036899540573358536, -0.014048400335013866, 0.004546699114143848, 0.06626969575881958, -0.010712770745158195, 0.003231824142858386, -0.014574350789189339, 0.03343934565782547, -0.011480933986604214, -0.0035847641993314028, 0.012989579699933529, 0.0023512039333581924, 0.030948003754019737, 0.009847721084952354, 0.04520401358604431, 0.02271273359656334, -0.03133554384112358, -0.019003402441740036, 0.025079509243369102, 0.009204124100506306, -0.042601946741342545, -0.015598569065332413, 0.010809656232595444, 0.019916893914341927, 0.012809650041162968, -0.008512085303664207, 0.026712721213698387, 0.05237354338169098, 0.015127982012927532, -0.007764682173728943, -0.014989574439823627, 0.006076106335967779, -0.04628359526395798, -0.019916893914341927, 0.0012249097926542163, -0.02149474434554577, 0.030505098402500153, 0.0044567338190972805, 0.015570887364447117, -0.006172991823405027, -0.019031085073947906, -0.012415187433362007, -0.01376466453075409, 0.031141774728894234, 0.020221391692757607, 0.02441515028476715, -0.013806186616420746, -0.02856738679111004, -0.022781938314437866, -0.011217959225177765, 0.0008858104702085257, -0.006844270043075085, -0.021453222259879112, -0.02106568031013012, -0.039612337946891785, -0.03429747372865677, -0.04916248098015785, -0.02301723137497902, -0.0364566370844841, -0.009930766187608242, -0.0004640989354811609, -0.00206573773175478, 0.04027669504284859, -0.026200613006949425, -0.02737708017230034, 0.012768127024173737, -0.0035882245283573866, -0.04307253286242485, -0.010373670607805252, 0.010823496617376804, -0.015155663713812828, 0.010415193624794483, -0.004986144136637449, -0.003993067424744368, -0.02600684203207493, 0.04002755880355835, -0.00045198824955150485, 0.02895492874085903, -0.01880963146686554, -0.028761157765984535, -0.02408297173678875, -0.0030501638539135456, -0.03410370275378227, -0.015446320176124573, 0.0035605428274720907, 0.001130619435571134, -0.0245120357722044, -0.001615046989172697, 0.0295085608959198, -0.03338398039340973, -0.024775011464953423, -0.03606909513473511, 0.011266401968896389, -0.030948003754019737, -0.006660879589617252, 0.003145319176837802, -0.0038719605654478073, -0.0060691856779158115, 0.03468501567840576, 0.008325234055519104, -0.0013365010963752866, -0.004048430826514959, 0.0014766391832381487, -0.03366079926490784, -0.003920403309166431, -0.0167058315128088, 0.019377103075385094, -0.00643942691385746, -0.005038047209382057, 0.030006829649209976, -0.033633116632699966, -0.01116951648145914, 0.013501688838005066, 0.011134914122521877, 0.019335580989718437, 0.004415211733430624, -0.007778523024171591, 0.0631139948964119, -0.034546609967947006, -0.019280217587947845, 0.0006107247900217772, -0.0023581243585795164, 0.020553570240736008, 0.028484342619776726, 0.023557022213935852, 0.02139785885810852, -0.01795150339603424, 0.04155004769563675, 0.02124560996890068, -0.008103781379759312, 0.012249098159372807, -0.0023563941940665245, -0.0008615890983492136, -0.006065725348889828, 0.020719660446047783, 0.025439368560910225, -0.0065812948159873486, 0.010726611129939556, 0.046532731503248215, -0.0004623688291758299, -0.01116951648145914, -0.0016764655010774732, -0.025868434458971024, -0.009287169203162193, -0.015446320176124573, -9.672332089394331e-05, -0.012899614870548248, 0.017702369019389153, -0.016345972195267677, 0.035681553184986115, -0.03518328443169594, 0.008456721901893616, 0.014546669088304043, 0.018892675638198853, 0.016484379768371582, -0.01150861568748951, -0.00029736070428043604, -0.020276755094528198, 0.01128024235367775, 0.018878836184740067, 0.01077505387365818, 0.034159064292907715, -0.003192031756043434, -0.03543241694569588, 0.028428979218006134, 0.0018909977516159415, -0.009543223306536674, 0.0023754253052175045, 0.014975733123719692, -0.04088569059967995, -0.00874737836420536, -0.0017889218870550394, -0.01939094439148903, 0.044124435633420944, -0.007868488319218159, 0.020041462033987045, -0.024179857224225998, 0.0475015863776207, -0.008532846346497536, -0.00014057051157578826, -0.003937704488635063, 0.008664333261549473], index=0, object='embedding'), Embedding(embedding=[-0.012739777565002441, 0.016879824921488762, 0.04386623576283455, -0.023348648101091385, -0.010517545975744724, -0.028843343257904053, 0.03756484016776085, 0.011187260039150715, -0.03783881291747093, 0.01519032008945942, 0.055251363664865494, -0.05403370410203934, -0.0031392821110785007, 0.0014298002934083343, 0.0045700338669121265, -0.00034960187622345984, -0.014695645309984684, 0.04971100762486458, -0.023287765681743622, 0.031233003363013268, 0.028401941061019897, -0.005601240321993828, -0.03223757445812225, 0.017823511734604836, -0.006042642518877983, -0.01815836876630783, 0.010411000810563564, 0.03156786039471626, 0.04121782258152962, -0.0013584529515355825, 0.008965028449892998, -0.03960442170500755, -0.0042656185105443, -0.026316696777939796, -0.022785481065511703, 0.011468843556940556, 0.025920957326889038, 0.007853913120925426, -0.0036282490473240614, -0.039878394454717636, -0.011537337675690651, -0.02697118930518627, 0.014079204760491848, 0.005761058069765568, 0.025677425786852837, 0.03126344457268715, 0.011187260039150715, -0.025327347218990326, -0.004726046696305275, 0.07208552956581116, -0.00031107431277632713, 0.048980411142110825, -0.028173629194498062, 0.043592263013124466, -0.001961575588211417, -0.05936096981167793, -0.03863029181957245, 0.04337916895747185, 0.035190399736166, -0.05321178212761879, 0.08481008559465408, -0.021339507773518562, 0.03668203577399254, 0.016803720965981483, -0.03628629446029663, 0.006925446446985006, -0.019726106896996498, 0.012853933498263359, 0.03820411115884781, 0.03030453622341156, -0.015197930857539177, -0.003725281450897455, -0.013409490697085857, -0.009025911800563335, -0.06015244871377945, -0.0031887495424598455, -0.008333367295563221, 0.02053280733525753, 0.02512947842478752, -0.037930138409137726, 0.01410964597016573, -0.020152287557721138, 0.016681954264640808, -0.04785407334566116, -0.038995590060949326, -0.023698726668953896, -0.02726038359105587, 0.034612011164426804, -0.0038128008600324392, -0.025646982714533806, -0.02088288590312004, 0.013744347728788853, -0.015091384761035442, 0.004714631009846926, 0.007793029770255089, -0.008447522297501564, 0.025753527879714966, -0.03522084280848503, -0.02368350513279438, -0.0007819666061550379, 0.059878475964069366, -0.009885884821414948, -0.02066979371011257, 0.03817367181181908, 0.02077633887529373, -0.027914876118302345, -0.053546641021966934, 0.021339507773518562, -0.04143091291189194, -0.014079204760491848, -0.051385290920734406, -0.008302925154566765, -0.009246612899005413, -0.007439147215336561, 0.04608846455812454, 0.011940687894821167, 0.0577780120074749, 0.026438463479280472, 0.022115767002105713, 0.06283130496740341, -0.019178159534931183, 0.004710825625807047, -0.007998510263860226, 0.007815861143171787, -0.0319940410554409, -0.010220741853117943, 0.027108175680041313, -0.009337937459349632, 0.015662163496017456, -0.012694114819169044, -0.021324286237359047, -0.01053276751190424, -0.0006154895527288318, 0.023135557770729065, -0.0003612552536651492, -0.06508398056030273, -0.019025951623916626, -0.0021917896810919046, -0.012351647950708866, -0.006818901281803846, 0.038386762142181396, -0.04024369269609451, 0.031415652483701706, 0.007484809495508671, -0.022146208211779594, -0.0038660734426230192, -0.07318142056465149, 0.0031716262456029654, -0.007785419467836618, -0.010433832183480263, -0.05957406014204025, -0.0086758341640234, -0.07677352428436279, 0.013013751246035099, 0.02838672138750553, -0.029604380950331688, -0.07159846276044846, 0.025220802053809166, -0.032024484127759933, 0.051811471581459045, -0.043988000601530075, -0.043774910271167755, 0.022222312167286873, 0.007431536912918091, 0.02523602358996868, -0.004996214993298054, -0.01631665602326393, 0.004988604690879583, -0.052055004984140396, -0.05281604453921318, -0.0286454726010561, 3.754652789211832e-05, -0.026468904688954353, -0.07829559594392776, -0.008135497570037842, -0.07184199243783951, -0.06788459420204163, -0.05129396542906761, -0.015251203440129757, -0.004988604690879583, 0.05321178212761879, 0.04645376652479172, -0.006731381639838219, 0.016621071845293045, -0.026818981394171715, -0.07409466803073883, 0.03135477006435394, 0.01589047536253929, -0.0037956773303449154, 0.02042626217007637, 0.008363808505237103, -0.022892026230692863, 0.042435482144355774, 0.02414012886583805, 0.050289396196603775, -0.09552550315856934, 0.052663836628198624, -0.01783873327076435, 0.024977270513772964, -0.006624836474657059, 0.000480643124319613, 0.0259818397462368, 0.005936096888035536, 0.03765616565942764, -0.00993154663592577, -0.02325732447206974, -0.012389699928462505, 0.01318878959864378, -0.016057902947068214, -0.023105116561055183, 0.02832583710551262, 0.002511425642296672, -0.010304455645382404, -0.0007139488589018583, -0.057260505855083466, 0.008226822130382061, -0.020913327112793922, 0.048949968069791794, -0.004334111697971821, -0.022283194586634636, 0.030471965670585632, -0.010190299712121487, 0.025966620072722435, 0.03917824104428291, -0.039239123463630676, 0.035190399736166, 0.04502301290631294, 0.011598220095038414, 0.03656027093529701, -0.03954353928565979, 0.02343997359275818, 0.010768689215183258, -0.01933036744594574, -0.0792088434100151, -0.01575348898768425, 0.023242102935910225, -0.010913286358118057, -0.002172763692215085, -0.0032115806825459003, -0.0024029777850955725, -0.06922402232885361, 0.011347077786922455, 0.047488775104284286, 0.017595199868083, 0.0286454726010561, -0.020045742392539978, -0.017321227118372917, -0.04386623576283455, 0.0017313616117462516, -0.0010806741192936897, 0.056803882122039795, -0.00973367691040039, -0.012754998169839382, -0.00748861487954855, -0.03586011379957199, 0.023881375789642334, 0.006297590211033821, 0.005152227822691202, 0.026347137987613678, 0.019406471401453018, -0.03522084280848503, 0.05652990937232971, -0.04392711818218231, 0.034398920834064484, -0.00878237932920456, -0.01684938371181488, 0.05086778476834297, 0.024688076227903366, 0.03467289358377457, 0.01805182360112667, -0.012032012455165386, -0.0011130182538181543, 0.023805271834135056, -0.002941412152722478, -0.0012794953072443604, -0.007328796666115522, -0.006807485595345497, 0.019695665687322617, 0.008614950813353062, 0.00949014537036419, -0.02315077930688858, -0.0025685036089271307, 0.02031971700489521, -0.04697126895189285, 0.04946747422218323, -0.05869125574827194, 0.0525420680642128, -0.02566220425069332, 0.04867599532008171, 0.02191789634525776, 0.028965109959244728, -0.018219251185655594, 0.008082224056124687, -0.013919387012720108, 0.007439147215336561, -0.023835713043808937, -0.0023820491041988134, -0.03324214369058609, 0.018067043274641037, 0.024703295901417732, 0.03716909885406494, 0.00750383548438549, 0.007248887792229652, 0.01508377492427826, -0.022998571395874023, 0.01933036744594574, -0.024825062602758408, -0.05025895684957504, 0.021019872277975082, 0.025038152933120728, 0.03908691555261612, 0.023698726668953896, -0.011484065093100071, -0.03719954192638397, -0.02400314062833786, -0.08474919945001602, 0.01855410821735859, 0.015844812616705894, 0.042465925216674805, 0.0173516683280468, 0.03646894544363022, -0.013363828882575035, 0.030669834464788437, -0.004589059855788946, -0.014383619651198387, 0.013006140477955341, 0.013447542674839497, -0.04322696477174759, 0.05500783398747444, 0.016758058220148087, -0.009170508943498135, -0.029969679191708565, -0.02453586831688881, -0.005022851284593344, 0.012351647950708866, -0.005852383095771074, 0.0017827317351475358, -0.0064117456786334515, 0.0241857897490263, -0.022983349859714508, -0.06295306980609894, -0.033150818198919296, 0.025936177000403404, -0.02429233491420746, 0.02159826084971428, 0.015357748605310917, 0.013401880860328674, 0.02691030688583851, -0.0012005375465378165, 0.01499245036393404, 0.02414012886583805, -0.005898045375943184, 0.0064003304578363895, 0.013850892893970013, 0.058356400579214096, -0.062009382992982864, 0.03318126127123833, -0.025586100295186043, -0.0036263465881347656, 0.016164448112249374, 0.014010711573064327, 0.013614971190690994, 0.042435482144355774, 0.03774748742580414, -0.05260295420885086, 0.023622622713446617, 0.012739777565002441, -0.028508486226201057, 0.03698645159602165, 0.027169059962034225, 0.055038273334503174, -0.03802146390080452, -0.007975678890943527, -0.013340997509658337, -0.06076128035783768, 0.04429241642355919, -0.052663836628198624, 0.02768656611442566, -0.024718517437577248, 0.02686464414000511, 0.003169723553583026, -0.03147653490304947, -0.04283122345805168, -0.021902676671743393, -0.02130906656384468, 0.028554148972034454, -0.032054923474788666, 0.018249692395329475, -0.01085240300744772, -0.02312033623456955, 0.005859993398189545, 0.00012937647989019752, 0.059665385633707047, 0.006952082738280296, 0.011605830863118172, 0.02194833755493164, 0.00530063034966588, 0.0687369629740715, -0.01695592887699604, 0.012960478663444519, 0.04240504279732704, 0.012032012455165386, -0.00015280218212865293, 0.008302925154566765, -0.03009144589304924, 0.05911744013428688, -0.019208600744605064, -0.00946731399744749, 0.04414020851254463, 0.02066979371011257, 0.03232889622449875, 0.02095898799598217, 0.04675817862153053, -0.009391210041940212, 0.019071614369750023, 0.01621011085808277, -0.010304455645382404, 0.04286166653037071, -0.004056333098560572, 0.053577080368995667, 0.008576898835599422, -0.060304656624794006, -0.04523610323667526, 0.019665224477648735, -0.03689512610435486, 0.04974145069718361, -0.004657553043216467, -0.0320853665471077, -0.06587545573711395, -0.041613563895225525, -0.030030563473701477, 0.017519095912575722, 0.019954418763518333, -0.012853933498263359, 0.05665167421102524, -0.045966699719429016, -0.002121393568813801, 0.04480992257595062, -0.000500382564496249, -0.04840202257037163, 0.03622541204094887, 0.007226056419312954, 0.007758783176541328, -0.01567738503217697, -0.038782499730587006, 0.04228327423334122, -0.025266464799642563, -0.011925466358661652, 0.05260295420885086, -0.034003183245658875, -0.04645376652479172, -0.04505345597863197, -0.0010159858502447605, -0.02718427963554859, 0.028447603806853294, -0.057260505855083466, 0.008211600594222546, -0.008949807845056057, 0.05866081640124321, -0.0129909198731184, 0.023272544145584106, 0.0013194497441872954, -0.00319445738568902, -0.034398920834064484, -0.01812792755663395, 0.027884434908628464, -0.003002295270562172, 0.00031226343708112836, 0.02942173182964325, -0.011438402347266674, -0.0005669733509421349, -0.03372920677065849, 0.0031792365480214357, 0.026423241943120956, -0.00949014537036419, -0.007404900621622801, -0.03738218918442726, 0.03342479094862938, -0.015951357781887054, 0.005551772657781839, 0.03406406566500664, -0.004303670488297939, -0.017199460417032242, 0.02400314062833786, -0.027747448533773422, 0.10076144337654114, 0.03115689940750599, -0.02726038359105587, 0.0018464686581864953, -0.02213098667562008, -0.011407961137592793, 0.02506859414279461, -0.008295315317809582, -0.019771769642829895, -0.008706275373697281, 0.01520554069429636, -0.014611931517720222, -0.00543381180614233, 0.02474895864725113, -0.016270995140075684, 0.009710845537483692, -0.012214661575853825, 0.021156858652830124, 0.004280839115381241, 0.02477939985692501, 0.023272544145584106, -0.03981751203536987, 0.010395780205726624, 0.009566248394548893, 0.03756484016776085, -0.0356774665415287, -0.04404888302087784, -0.02208532579243183, -0.03327258676290512, -0.02117208018898964, 0.04806716367602348, 0.021857013925909996, 0.010380559600889683, -0.03765616565942764, 0.023424752056598663, -0.013074634596705437, 0.03939133137464523, -0.03908691555261612, -0.02651456743478775, 0.035281725227832794, -0.03656027093529701, -0.0036529828794300556, -0.09156810492277145, -0.02138517051935196, 0.011887415312230587, 0.04502301290631294, 0.009056353010237217, 0.012298375368118286, 0.004897280130535364, -0.017853952944278717, -0.0320853665471077, -0.0014383619418367743, -0.005700175184756517, -0.013432322070002556, 0.0012661771615967155, -0.044657714664936066, 0.0005365318502299488, 0.0170624740421772, 0.013972659595310688, -0.024794621393084526, 0.03263331204652786, 0.014984839595854282, 0.008873703889548779, 0.010022871196269989, 0.018614990636706352, 0.013272504322230816, 0.03762572258710861, 0.0020643158350139856, -0.019299926236271858, -0.010068533942103386, -0.018188809975981712, -0.009079184383153915, -0.020806781947612762, 0.012808270752429962, 0.014703256078064442, 0.010372948832809925, 0.018508445471525192, -0.0022659909445792437, -0.04590581730008125, -0.03153741732239723, 0.008264873176813126, 0.027321267873048782, 0.0016447935486212373, 0.0018835692899301648, -0.013866113498806953, -0.022207090631127357, -0.037229981273412704, -0.013181179761886597, -0.027351709082722664, -0.04888908565044403, -0.023774830624461174, 0.0007995656342245638, 0.012397310696542263, 0.001808416796848178, -0.04435329884290695, -0.02039582096040249, -0.029969679191708565, -0.005734421778470278, 0.009277054108679295, 0.01762564294040203, 0.009474923834204674, 0.027427813038229942, 0.015692604705691338, -0.0173516683280468, -0.012998530641198158, 0.017001591622829437, -0.006396525073796511, 0.015388189814984798, 0.011240532621741295, 0.04386623576283455, -0.01684938371181488, -0.00038551335455849767, -0.039239123463630676, -0.0009113430860452354, -0.014376009814441204, 0.008188770152628422, 0.005715395789593458, -0.00420473562553525, -0.016621071845293045, -0.0017342155333608389, 0.0021879845298826694, -0.007914796471595764, -0.0033333469182252884, 0.016879824921488762, -0.010631701909005642, 0.0038870021235197783, -0.01762564294040203, -0.013683464378118515, -0.030959028750658035, 0.02824973315000534, -0.014299905858933926, -0.02325732447206974, -0.01989353634417057, 0.005837162025272846, 0.0006349911564029753, 0.00030084786703810096, 0.016803720965981483, -0.01876719854772091, -0.03062417171895504, -0.007659848313778639, 0.0436227023601532, 0.0023382895160466433, -0.05643858388066292, 0.022313635796308517, 0.01044144295156002, -0.004254202824085951, -0.0029737562872469425, -0.0011434596963226795, -0.04648420587182045, -0.03933044895529747, 0.02538822963833809, -0.011263363994657993, 0.028904225677251816, 0.015951357781887054, 0.0014849755680188537, 0.04785407334566116, -0.017123356461524963, 0.006187239661812782, -0.022207090631127357, 0.03933044895529747, -0.008340977132320404, 0.01401832140982151, -0.023942258208990097, 0.01492395717650652, -0.02191789634525776, -0.02213098667562008, 0.015365359373390675, 0.019634783267974854, -0.037473514676094055, 0.0031373794190585613, -0.011004610918462276, -0.027001630514860153, -0.019102055579423904, -0.006476433947682381, -0.026210151612758636, 0.015159878879785538, -0.01642320118844509, 0.017153797671198845, -0.03187227621674538, 0.009969598613679409, 0.01086762361228466, 0.006343252491205931, 0.0019653807394206524, -0.020380599424242973, -0.005654512904584408, -0.003871781285852194, 0.05175058916211128, 0.03287684544920921, 0.02229841612279415, -0.015479514375329018, 0.014695645309984684, -0.018538888543844223, -0.007671264000236988, -0.0388738252222538, 0.016651513054966927, -0.02488594502210617, 0.05229853838682175, -0.016986370086669922, 0.0010454760631546378, 0.005692564882338047, -0.01783873327076435, 0.015555618330836296, -0.0150989955291152, 0.014535827562212944, 0.04319652169942856, -0.021613482385873795, -0.01596657931804657, -0.021583039313554764, 0.01631665602326393, 0.03254199028015137, -0.07178111374378204, -0.008333367295563221, 0.014490164816379547, -0.01695592887699604, 0.03652982786297798, -0.022435402497649193, 0.0021765688434243202, -0.014284685254096985, -0.014688035473227501, -0.04477947950363159, 0.0004454450972843915, 0.01954345777630806, 0.059178322553634644, 0.030471965670585632, 0.028067084029316902, -0.007412510924041271, 0.0005298727774061263, 0.030243653804063797, 0.011955908499658108, -0.02293768711388111, 0.004109605681151152, -0.01578393019735813, -0.04000016301870346, -0.0014336054446175694, -0.035738348960876465, 0.0016638195374980569, 0.0013356218114495277, -0.01292242668569088, -0.03135477006435394, -0.003076496534049511, -0.02071545645594597, 0.0373213067650795, -0.04286166653037071, -0.007458173204213381, 0.021963559091091156, 0.021720027551054955, 0.012793050147593021, -0.014809801243245602, -0.018919406458735466, -0.017869174480438232, -0.00214232224971056, -0.022770259529352188, 0.03476421907544136, -0.013219231739640236, -0.0341249480843544, 0.006229096557945013, -0.01564694382250309, 0.006556343287229538, 0.008881314657628536, -0.017686525359749794, 0.001699017477221787, 0.05369884893298149, 0.0027016852982342243, -0.018508445471525192, -0.004512955900281668, -0.029330408200621605, 0.01535013783723116, -0.007313576061278582, 0.01407159399241209, 0.04173532873392105, -0.03692556917667389, 0.015601281076669693, -0.02669721655547619, 0.013401880860328674, 0.0008742425125092268, 0.049406591802835464, 0.016910266131162643, -0.0022488676477223635, -0.002992782276123762, 0.014817411080002785, 0.01695592887699604, -0.004935332108289003, 0.009033521637320518, 0.000438786024460569, 0.025038152933120728, -0.03266375511884689, -0.014200970530509949, 0.025905735790729523, 0.004090579692274332, 0.03360744193196297, -0.007682679686695337, 8.524518307240214e-06, 0.016681954264640808, -0.00644218735396862, 0.028188850730657578, -0.039421774446964264, -0.018356239423155785, -0.015487125143408775, -0.002867210889235139, -0.018310576677322388, 0.03296817094087601, -0.026286255568265915, 0.026742877438664436, -0.005308240652084351, -0.04188753664493561, 0.019299926236271858, 0.02640802226960659, 0.014147697947919369, 0.024170570075511932, -0.01607312448322773, -0.002851990284398198, -0.004634722135961056, -0.0751296803355217, 0.027975760400295258, -0.0016067416872829199, -0.008333367295563221, -0.027990980073809624, -0.019923977553844452, -0.02053280733525753, 0.052450746297836304, 0.02595139853656292, 0.006742797326296568, 0.05948273837566376, -0.00971845630556345, -0.01607312448322773, 0.00093940639635548, -0.007077654357999563, -0.008546457625925541, 0.01887374371290207, 0.0069368621334433556, -0.013234452344477177, -0.052876926958560944, -0.015951357781887054, -0.01210811547935009, -0.03214624896645546, 0.029756588861346245, -0.016681954264640808, 0.015631722286343575, -0.022496284916996956, 0.01369868591427803, -0.010152247734367847, -0.0024543479084968567, 0.022678934037685394, 0.0034398920834064484, 0.02117208018898964, -0.003630151739344001, -0.016438422724604607, -0.008356197737157345, -0.012321206741034985, -0.005030461587011814, 0.03981751203536987, 0.02240496128797531, 0.007937626913189888, -0.02793009765446186, -0.008485574275255203, 0.02552521787583828, -0.021430833265185356, 0.006145382300019264, 0.01054798811674118, -0.010829571634531021, -0.0016248163301497698, -0.029756588861346245, -0.026605891063809395, -0.007869133725762367, 0.017016811296343803, -0.01628621481359005, 0.019726106896996498, 0.027762670069932938, -0.010837182402610779, 0.008287704549729824, 0.02584485337138176, 0.029269523918628693, -0.014543437398970127, 0.03808234632015228, -0.028417162597179413, 0.006419356446713209, 0.023774830624461174, -0.04773230850696564, -0.02980225160717964, 0.03534260764718056, -0.014429282397031784, 0.013310556299984455, 0.02031971700489521, -0.011643882840871811, 0.014368399046361446, -0.010730637237429619, 0.007439147215336561, -0.004935332108289003, -0.029223863035440445, 0.017016811296343803, -0.0005422396352514625, -0.01256473921239376, 0.008660613559186459, 0.03631673753261566, -0.002408685628324747, 0.016590630635619164, -0.019984859973192215, 0.008554068394005299, -0.005681149195879698, -0.009155288338661194, -0.006632446777075529, 0.024474984034895897, 0.02902599237859249, -0.019589120522141457, -0.009642352350056171, 0.02173524722456932, -0.0067618233151733875, 0.002737834583967924, -0.01869109459221363, 0.03238978236913681, 0.03016754984855652, 0.009315106086432934, 0.005852383095771074, 0.020517587661743164, -0.0015791540499776602, -0.03022843226790428, 0.019406471401453018, -0.02088288590312004, 0.04477947950363159, 0.01401832140982151, 0.021720027551054955, -0.02672765776515007, 0.014543437398970127, -0.010342507623136044, 0.04474904015660286, 0.029756588861346245, -0.001847420004196465, 0.06207026541233063, 0.016864603385329247, -0.010296844877302647, 0.008630171418190002, -0.005703980568796396, -0.0002646985522005707, 0.005369123537093401, -0.0014821216464042664, 0.014147697947919369, -0.019056392833590508, 0.01756475865840912, -0.043805353343486786, 0.05263339355587959, 0.007545692380517721, -0.015175099484622478, -0.020837223157286644, 0.002787302015349269, -0.040061045438051224, -0.04590581730008125, 0.06264865398406982, -0.00418570963665843, 0.008812821470201015, -0.04389667510986328, -0.022572388872504234, 0.01866065338253975, -0.007085264660418034, 0.006179629359394312, 0.02085244283080101, 0.00743534229695797, -0.02595139853656292, -0.02477939985692501, -0.014771749265491962, 0.008242042735219002, -0.03668203577399254, -0.035920996218919754, 0.015707826241850853, -0.02039582096040249, -0.01908683590590954, 0.015281644649803638, -0.03939133137464523, -0.004067748785018921, 0.005890434607863426, 0.014277074486017227, -0.01946735382080078, 0.0062519279308617115, -0.002313555683940649, 0.030715497210621834, 0.0209894310683012, 0.00654112221673131, 0.002743542194366455, -0.03366832435131073, -0.019771769642829895, -0.006012200843542814, 0.00011302604980301112, 0.011423181742429733, -0.016255773603916168, -0.02407924458384514, 0.02382049150764942, 0.04383579269051552, 0.028021423146128654, 0.015418631955981255, 0.02031971700489521, 0.0009660427458584309, 0.01304419245570898, 0.0315069779753685, 0.013614971190690994, -0.0043683587573468685, -0.005799110047519207, 0.006902615539729595, -0.04277034103870392, 0.0030099055729806423, -0.0007676971727050841, 0.012260323390364647, 0.005228331778198481, 0.014330347068607807, 0.02031971700489521, -0.010936117731034756, -0.0259818397462368, 0.02528168447315693, -0.007716926280409098, -0.02456630952656269, 0.013219231739640236, -0.006860758177936077, -0.011225312016904354, 4.2213832784909755e-05, -0.012389699928462505, -0.02640802226960659, -0.015091384761035442, -0.01578393019735813, -0.029436953365802765, -0.0016752351075410843, -0.027701785787940025, -0.026210151612758636, 0.03649938479065895, 0.036803800612688065, -0.047701865434646606, 0.008805210702121258, -0.009391210041940212, 0.008554068394005299, -0.005605045706033707, -0.010715416632592678, -0.01515226811170578, 0.007724536582827568, 0.016407981514930725, -0.0007700754213146865, -0.035981882363557816, -0.00535390293225646, -0.010350118391215801, 0.02726038359105587, 0.010677364654839039, -0.021293845027685165, -0.013812840916216373, -0.0393608883023262, -0.009132456965744495, 0.003660593181848526, 0.0002984696184284985, -0.0003431806107982993, 0.020456703379750252, 0.01663629338145256, 0.007899574935436249, -0.006248122546821833, -0.030608952045440674, 0.0035255090333521366, 0.0020700236782431602, 0.019589120522141457, 0.03957397863268852, 0.015426241792738438, -0.01783873327076435, -0.02173524722456932, 0.03327258676290512, -0.01187980454415083, -0.035099077969789505, 0.023424752056598663, 0.008789990097284317, 0.020441483706235886, 0.03129388764500618, 0.001163436914794147, 0.020152287557721138, -0.019315145909786224, -0.013173568993806839, -0.006955888122320175, 0.02149171568453312, 0.02640802226960659, 0.009551027789711952, 0.0017589492490515113, -0.013614971190690994, -0.006723771337419748, 0.03254199028015137, 0.003624443896114826, 0.022633273154497147, -0.0004085823311470449, -0.01780829206109047, 0.010045702569186687, -0.003858463140204549, -0.026879865676164627, -0.03372920677065849, 0.02347041480243206, 0.001680942834354937, 0.010106585919857025, -0.041583120822906494, 0.028280174359679222, 0.016971148550510406, 0.001777975237928331, 0.014581489376723766, -0.013607361353933811, -0.00431128079071641, 0.0073440177366137505, -0.03738218918442726, -0.02608838491141796, -0.021156858652830124, -0.001469754846766591, -0.01385850366204977, 0.0005317753530107439, 0.0032629508059471846, 0.007564718369394541, -0.0033599832095205784, -0.015578449703752995, 0.022222312167286873, -0.00444446224719286, -0.02184179238975048, -0.03001534193754196, -0.030000120401382446, 0.028295395895838737, -0.003923151176422834, -0.02715383842587471, -0.031096016988158226, 0.021522156894207, 0.04913261905312538, 0.006556343287229538, -0.03926956653594971, 0.013379049487411976, -0.010913286358118057, -0.013074634596705437, 0.0149772297590971, -0.027884434908628464, 0.04118737950921059, 0.00378426187671721, 0.03607320412993431, -0.02397269941866398, -0.016940707340836525, 0.008401860482990742, 0.020274054259061813, 0.016225332394242287, -0.028127968311309814, -0.001984406728297472, -0.014170529320836067, 0.01957389898598194, -0.018508445471525192, -0.006232901941984892, 0.004121021367609501, -0.03351611644029617, -0.002509523183107376, -0.004870643839240074, 0.01405637338757515, 0.04155267775058746, 0.018112706020474434, -0.0015163683565333486, -0.004547202493995428, 0.006940667517483234, -0.014193360693752766, 0.019802210852503777, -0.010076143778860569, -0.02283114194869995, -0.0016086442628875375, 0.009010691195726395, 0.0194369126111269, 0.007298355456441641, 6.801777635701001e-05, -0.02007618546485901, 0.02803664281964302, 0.0037690410390496254, 0.009695624932646751, 0.011978739872574806, -0.009497755207121372, -0.008120276033878326, 0.09942201524972916, -6.314237543847412e-05, -0.054825183004140854, -0.010646922513842583, -0.015228372067213058, 0.005190279800444841, -0.0055137211456894875, -0.012998530641198158, -0.04216150939464569, -0.018432341516017914, 0.01911727711558342, -0.0061986553482711315, 0.015981798991560936, 0.007735952269285917, 0.02333342842757702, 0.022100545465946198, -0.02945217303931713, 0.031811390072107315, 0.019558679312467575, -0.004151462577283382, 0.00869105476886034, 0.014246633276343346, 0.009231392294168472, -0.011484065093100071, -0.0073972903192043304, -0.01639275997877121, -0.0029737562872469425, 0.003281976794824004, 0.016879824921488762, 0.01567738503217697, 0.014558658935129642, -0.040913406759500504, -0.000190854087122716, -0.012024401687085629, 0.0037462098989635706, 0.004665163345634937, -0.0233638696372509, 0.004680384416133165, -0.007275524083524942, -0.03753439709544182, -0.021765688434243202, 0.041796211153268814, -0.006902615539729595, -0.007229861803352833, 0.032054923474788666, -0.0021309065632522106, 0.024946829304099083, 0.020624132826924324, -0.032268013805150986, -0.017306005582213402, 0.018843302503228188, 0.028051864355802536, -0.006563953589648008, 0.015319696627557278, -0.012100505642592907, -0.048128049820661545, -0.012587569653987885, -0.002962340833619237, 0.029467394575476646, 0.03385097533464432, -0.0304415225982666, 0.016758058220148087, 0.04648420587182045, -0.016240552067756653, -0.004482514224946499, -0.02739737182855606, -0.008736717514693737, 0.012214661575853825, -0.026651553809642792, 0.011522116139531136, 0.016240552067756653, 0.015586059540510178, -0.06277041882276535, -0.01586003415286541, -0.012077674269676208, 0.00014364594244398177, -0.017214681953191757, -0.03719954192638397, 0.018965069204568863, 0.032937727868556976, 0.018736757338047028, -0.0035312166437506676, -0.006933056749403477, 0.007366848643869162, 0.03540349006652832, 0.002926191547885537, -0.0087975999340415, -0.016027461737394333, 0.013828062452375889, 0.007401095237582922, -0.015380579978227615, -0.028828121721744537, 0.014589100144803524, 0.004181904252618551, 0.012344038113951683, -0.029863134026527405, -0.04417065158486366, 0.007324991747736931, -0.00971845630556345, -0.02251150645315647, 0.010045702569186687, -0.01898029074072838, -0.028736798092722893, 0.005981759168207645, -0.004303670488297939, -0.011544947512447834, 0.0030650808475911617, 0.026849424466490746, -0.022420182824134827, 0.019741328433156013, -0.007686484605073929, -0.013234452344477177, 0.0194369126111269, -0.026895085349678993, 0.003888904582709074, -0.035981882363557816, -0.01308224443346262, 0.010015261359512806, -0.0013422808842733502, -0.013204011134803295, 0.012602790258824825, 0.017686525359749794, -0.009619520977139473, 0.02120252139866352, -0.03187227621674538, 0.03808234632015228, -0.024170570075511932, -0.012747388333082199, 0.016697175800800323, -0.026788540184497833, -0.012100505642592907, 0.004414021037518978, 0.0218113511800766, 0.017549538984894753, 0.032054923474788666, 0.005859993398189545, 0.006636252161115408, -0.0109056755900383, 0.026240592822432518, 0.004874448757618666, -0.013295335695147514, -0.00016802293248474598, 0.0001567262806929648, -0.0013812840916216373, 0.023729167878627777, 0.010471884161233902, 0.007876744493842125, -0.012876763939857483, 0.01504572294652462, -0.009391210041940212, 0.015433852560818195, -0.009528196416795254, -0.031141677871346474, 0.03951309621334076, 0.00945209339261055, 0.031065573915839195, 0.04000016301870346, 0.0018845205195248127, -0.028736798092722893, -0.03070027567446232, 0.04602758213877678, -0.0031449899543076754, -0.03115689940750599, -0.01054798811674118, 0.023196440190076828, 0.02007618546485901, -0.03656027093529701, 0.010806741192936897, 0.015456683933734894, -0.00954341795295477, 0.0356774665415287, 0.014969618991017342, -0.004136241972446442, -0.014802190475165844, 0.029254304245114326, -0.01384328305721283, -0.011917856521904469, 0.029147759079933167, 0.023424752056598663, -0.0031887495424598455, -0.020867664366960526, 0.01617966964840889, 0.008203990757465363, 0.02566220425069332, -0.029436953365802765, 0.0034341842401772738, -6.659083010163158e-05, -0.004973384086042643, -0.008302925154566765, 0.030882924795150757, 0.010464273393154144, 0.008127886801958084, 0.009817391633987427, -0.028188850730657578, 0.019771769642829895, 0.016088346019387245, 0.027595240622758865, -0.00757613405585289, 0.015182709321379662, 0.017016811296343803, -0.017656084150075912, 0.005525136366486549, -0.007831081748008728, 0.008774769492447376, 0.00546044809743762, 0.00016397993022110313, -0.016986370086669922, -0.017306005582213402, -0.03070027567446232, -0.016149228438735008, 0.0619485005736351, 0.009946768172085285, -0.012968088500201702, -0.02347041480243206, 0.012496245093643665, 0.0036320541985332966, -0.016164448112249374, 0.012070064432919025, 0.008523626253008842, -0.003264853497967124, 0.03586011379957199, 0.01095894817262888, 0.04076119884848595, -0.002827256452292204, -0.007633212022483349, 0.029543498530983925, 0.014771749265491962, 0.003479846753180027, 0.0131355170160532, -0.021126417443156242, -0.004414021037518978, -0.027382150292396545, 0.012610401026904583, 0.011111156083643436, -0.0031659184023737907, -0.0009184778318740427, 0.0006240512011572719, -0.023409532383084297, 0.02496204897761345, 0.030669834464788437, 0.013226841576397419, -0.03540349006652832, -0.019710887223482132, -0.013386660255491734, 0.017153797671198845, -0.03671247884631157, 0.03238978236913681, -0.0152207612991333, -0.027351709082722664, 0.010137027129530907, 0.010593649931252003, -0.002323068678379059, -0.010662143118679523, -0.03847808390855789, -0.008751938119530678, 0.011309025809168816, -0.025220802053809166, -0.008356197737157345, 0.007907185703516006, -0.022648492828011513, -0.021994000300765038, -0.021324286237359047, -0.04295298829674721, 0.002374438801780343, -0.0041666836477816105, -0.01171998679637909, -0.008097445592284203, -0.009132456965744495, -0.024048803374171257, 0.035494815558195114, -0.028584590181708336, 0.0362558551132679, -0.0040449174121022224, 0.0023249713703989983, -0.029650043696165085, -0.000983641715720296, -0.003281976794824004, 0.04383579269051552, 0.01621011085808277, 0.03972618654370308, 0.041796211153268814, 0.00012414433876983821, 0.005962733179330826, -0.009817391633987427, 0.04371402785181999, -0.04520566388964653, 0.005064708646386862, 0.019208600744605064, 0.02173524722456932, -0.00017765483062248677, -0.009277054108679295, -0.011210091412067413, 0.0068150958977639675, -0.0043683587573468685, 0.04645376652479172, 0.028280174359679222, 0.059665385633707047, -0.03217669203877449, 0.01604268327355385, 0.018630212172865868, 0.005840967409312725, 0.0065335119143128395, -0.017960498109459877, 0.010989390313625336, -0.0025570879224687815, 0.014368399046361446, -0.003234411822631955, -0.01564694382250309, 0.02226797491312027, 0.04121782258152962, 0.007355432957410812, 0.007869133725762367, -0.016834162175655365, -0.006651472765952349, 0.003945982549339533, -0.02301379106938839, -0.027032073587179184, -0.009345547296106815, -0.05537313222885132, -0.00762940663844347, -0.009216170758008957, 0.06313572078943253, 0.007754978258162737, 0.01901073195040226, 0.021567819640040398, -0.027747448533773422, -0.011514506302773952, -0.021765688434243202, -0.0023192635271698236, -0.023135557770729065, 0.005810525733977556, 0.040517669171094894, 0.020806781947612762, 0.008706275373697281, 0.0286454726010561, 0.009779339656233788, 0.022998571395874023, -0.009596690535545349, 0.017716966569423676, -0.04800628125667572, 0.0032058728393167257, -0.014383619651198387, 0.003976423759013414, -0.045753609389066696, -0.005555578041821718, 0.01837145909667015, -0.02290724590420723, 0.0042465925216674805, 0.019208600744605064, -0.0046233064495027065, 0.017549538984894753, 0.003301002783700824, -0.010578429326415062, 0.023729167878627777, -0.02945217303931713, 0.017047252506017685, 0.043988000601530075, -0.014436892233788967, -0.024733737111091614, 0.007085264660418034, -0.002511425642296672, 0.01578393019735813, -0.0029870744328945875, -0.014528216794133186, -0.0011786577524617314, 0.011187260039150715, 0.026529787108302116, -0.009436871856451035, 0.0011272876290604472, 0.01639275997877121, -0.007614186033606529, -0.00643457705155015, -0.02566220425069332, 0.012298375368118286, 0.017123356461524963, 0.01261801179498434, -0.001627670251764357, -0.003949787467718124, -0.00968040432780981, -0.03981751203536987, 0.020197950303554535, -0.01837145909667015, -0.0007795884157530963, 0.014581489376723766, 0.04188753664493561, 0.006232901941984892, -0.017001591622829437, 0.019178159534931183, -0.006887394469231367], index=1, object='embedding')], model='text-embedding-3-small', object='list', usage=Usage(prompt_tokens=13, total_tokens=13))
print(f"vector 0: {len(res.data[0].embedding)}\nvector 1: {len(res.data[1].embedding)}")
vector 0: 1536
vector 1: 1536

# we can extract embeddings to a list
embeds = [record.embedding for record in res.data]
len(embeds)
2

Next, we initialize our index to store vector embeddings with Pinecone.

len(embeds[0])
1536
from pinecone import Pinecone

pc = Pinecone(api_key="...")
import time
from pinecone import ServerlessSpec

spec = ServerlessSpec(cloud="aws", region="us-west-2")

index_name = 'semantic-search-openai'

# check if index already exists (if shouldn't if this is your first run)
if index_name not in pc.list_indexes().names():
    # if does not exist, create index
    pc.create_index(
        index_name,
        dimension=len(embeds[0]),  # dimensionality of text-embed-3-small
        metric='dotproduct',
        spec=spec
    )
    # wait for index to be initialized
    while not pc.describe_index(index_name).status['ready']:
        time.sleep(1)

# connect to index
index = pc.Index(index_name)
time.sleep(1)
# view index stats
index.describe_index_stats()
{'dimension': 1536,
 'index_fullness': 0.0,
 'namespaces': {},
 'total_vector_count': 0}
from datasets import load_dataset

# load the first 1K rows of the TREC dataset
trec = load_dataset('trec', split='train[:1000]')
trec
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

Downloading data:   0%|          | 0.00/213k [00:00<?, ?B/s]
Downloading data:   0%|          | 0.00/17.1k [00:00<?, ?B/s]
Generating train split:   0%|          | 0/5452 [00:00<?, ? examples/s]
Generating test split:   0%|          | 0/500 [00:00<?, ? examples/s]
Dataset({
    features: ['text', 'coarse_label', 'fine_label'],
    num_rows: 1000
})
trec[0]
{'text': 'How did serfdom develop in and then leave Russia ?',
 'coarse_label': 2,
 'fine_label': 26}

Then we create a vector embedding for each phrase using OpenAI, and upsert the ID, vector embedding, and original text for each phrase to Pinecone.

from tqdm.auto import tqdm

count = 0  # we'll use the count to create unique IDs
batch_size = 32  # process everything in batches of 32
for i in tqdm(range(0, len(trec['text']), batch_size)):
    # set end position of batch
    i_end = min(i+batch_size, len(trec['text']))
    # get batch of lines and IDs
    lines_batch = trec['text'][i: i+batch_size]
    ids_batch = [str(n) for n in range(i, i_end)]
    # create embeddings
    res = client.embeddings.create(input=lines_batch, model=MODEL)
    embeds = [record.embedding for record in res.data]
    # prep metadata and upsert batch
    meta = [{'text': line} for line in lines_batch]
    to_upsert = zip(ids_batch, embeds, meta)
    # upsert to Pinecone
    index.upsert(vectors=list(to_upsert))
  0%|          | 0/32 [00:00<?, ?it/s]

Querying

With our data indexed, we're now ready to move onto performing searches. This follows a similar process to indexing. We start with a text query, that we would like to use to find similar sentences. As before we encode this with OpenAI's text similarity Babbage model to create a query vector xq. We then use xq to query the Pinecone index.

query = "What caused the 1929 Great Depression?"

xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

Now query...

res = index.query(vector=[xq], top_k=5, include_metadata=True)
res
{'matches': [{'id': '932',
              'metadata': {'text': 'Why did the world enter a global '
                                   'depression in 1929 ?'},
              'score': 0.751888752,
              'values': []},
             {'id': '787',
              'metadata': {'text': "When was `` the Great Depression '' ?"},
              'score': 0.597448647,
              'values': []},
             {'id': '400',
              'metadata': {'text': 'What crop failure caused the Irish Famine '
                                   '?'},
              'score': 0.367482603,
              'values': []},
             {'id': '835',
              'metadata': {'text': 'What were popular songs and types of songs '
                                   'in the 1920s ?'},
              'score': 0.324545294,
              'values': []},
             {'id': '262',
              'metadata': {'text': 'When did World War I start ?'},
              'score': 0.320995867,
              'values': []}],
 'namespace': '',
 'usage': {'read_units': 6}}

The response from Pinecone includes our original text in the metadata field, let's print out the top_k most similar questions and their respective similarity scores.

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")
0.75: Why did the world enter a global depression in 1929 ?
0.60: When was `` the Great Depression '' ?
0.37: What crop failure caused the Irish Famine ?
0.32: What were popular songs and types of songs in the 1920s ?
0.32: When did World War I start ?

Looks good, let's make it harder and replace "depression" with the incorrect term "recession".

query = "What was the cause of the major recession in the early 20th century?"

# create the query embedding
xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

# query, returning the top 5 most similar results
res = index.query(vector=[xq], top_k=5, include_metadata=True)

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")
0.63: Why did the world enter a global depression in 1929 ?
0.55: When was `` the Great Depression '' ?
0.34: What were popular songs and types of songs in the 1920s ?
0.33: What crop failure caused the Irish Famine ?
0.29: What is considered the costliest disaster the insurance industry has ever faced ?

And again...

query = "Why was there a long-term economic downturn in the early 20th century?"

# create the query embedding
xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

# query, returning the top 5 most similar results
res = index.query(vector=[xq], top_k=5, include_metadata=True)

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")
0.62: Why did the world enter a global depression in 1929 ?
0.54: When was `` the Great Depression '' ?
0.34: What were popular songs and types of songs in the 1920s ?
0.33: What crop failure caused the Irish Famine ?
0.32: What do economists do ?

Looks great, our semantic search pipeline is clearly able to identify the meaning between each of our queries and return the most semantically similar questions from the already indexed questions.

Once we're finished with the index we delete it to save resources.

pc.delete_index(index_name)